{"id":123523,"date":"2022-04-01T08:00:00","date_gmt":"2022-04-01T08:00:00","guid":{"rendered":"https:\/\/fedoramagazine.org\/?p=36107"},"modified":"2022-04-01T08:00:00","modified_gmt":"2022-04-01T08:00:00","slug":"using-sourcegraph-to-search-34000-fedora-repositories","status":"publish","type":"post","link":"https:\/\/sickgaming.net\/blog\/2022\/04\/01\/using-sourcegraph-to-search-34000-fedora-repositories\/","title":{"rendered":"Using Sourcegraph to Search 34,000+ Fedora Repositories"},"content":{"rendered":"

In October 2021, a Fedora Linux user asked a question about licensing<\/a>. Fedora Project Leader Matthew Miller left a response<\/a>: \u201cSince we don’t have a complete, exploded, searchable repository of all of the packages in Fedora, I don’t have a quick way to check.\u201d <\/p>\n

Followed by<\/a>: \u201c…or possibly pay Sourcegraph to do it for us. They seem like nice people.\u201d He is correct, we (Sourcegraph) are<\/em> nice people, but we don\u2019t want your money. Instead, we wanted to team up with the Fedora community.<\/p>\n

The Fedora Community can now search their universe of open source code\u2014currently over 34,000 repositories and counting.<\/p>\n

<\/span> <\/p>\n

Introduction to code search<\/h2>\n

For those who aren\u2019t familiar with the concept of code search<\/a>, it enables teams to onboard to a new codebase and find answers faster, helps to identify security risks, and many other use cases. Sourcegraph has indexed over two-million repositories across multiple code hosts such as GitHub and GitLab. This article is going to focus strictly on code search for src.fedoraproject.org<\/em>. <\/strong>Sourcegraph provides both a web app <\/a>and CLI<\/a> interface.<\/p>\n

Using the Web app<\/h2>\n

When using the Sourcegraph web app<\/a> you will need to start each search with repo:^src.fedoraprojects.org<\/strong> before entering any search queries. Using this link to the web app<\/a> will include this initial string as shown here:<\/p>\n

\"\"<\/a>
Sourcegraph web app interface<\/figcaption><\/figure>\n

The following sections will provide some web app examples of searches that might be of interest.<\/p>\n

Find repositories using popular OSI-approved licenses <\/h3>\n

The following query will scan all the repositories for software that is compatible with the “Open Source Definition” (OSD).<\/p>\n

repo:^src.fedoraproject.org\/ lang:\"RPM Spec\" License: ^.*apache|bsd|gpl|lgpl|mit|mpl|cddl|epl.*$<\/pre>\n
\"\"<\/a>
License search<\/figcaption><\/figure>\n
\n
Try it!<\/a><\/div>\n<\/div>\n
Find files with TODOs<\/h3>\n
The following query can find TODOs in 34k repositories. This is great for those looking to contribute to projects that need help.<\/p>\n
repo:^src.fedoraproject.org\/ \"TODO\"<\/pre>\n
\"\"<\/a>
Search for TODO<\/figcaption><\/figure>\n
\n
Try it!<\/a><\/div>\n<\/div>\n
Find files being served via FTP<\/h3>\n
A co-worker of mine from back in the day told me \u201cFTP is a dead protocol\u201d. Is it? You can add to this query to find any other protocol such as irc, https, etc.<\/p>\n
repo:^src.fedoraproject.org\/ (?:ftp):\/\/[A-Za-z0-9-]{0,63}(.[A-Za-z0-9-]{0,63})+(:d{1,4})?\/*(\/*[A-Za-z0-9-._]+\/*)*(?.*)?(#.*)?<\/pre>\n
\"\"<\/a>
Search for protocol<\/figcaption><\/figure>\n
\n
Try it!<\/a><\/div>\n<\/div>\n
Find files with a vulnerable version of Log4j<\/h3>\n
This query will find any files that are possibly vulnerable (false positives can happen) to CVE-2021-44228 aka Log4j. You can also search for other vulnerabilities that can then be reported to project maintainers.<\/p>\n
repo:^src.fedoraproject.org\/ org.apache.logging.log4j 2.((0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15)(.[0-9]+)) count:all<\/pre>\n
\"\"<\/a>
Search for log4j<\/figcaption><\/figure>\n
\n
Try it!<\/a><\/div>\n<\/div>\n
Use the CLI<\/h2>\n
Sourcegraph also has a command-line interface tool called src<\/a>, which allows you to do everything I just mentioned above, plus other useful commands like getting results in JSON for programmatic consumption.<\/p>\n
src search -json 'repo:^src.fedoraproject.org\/ lang:\"RPM Spec\" License: ^.*apache|bsd|g\npl|lgpl|mit|mpl|cddl|epl.*$'<\/pre>\n
JSON output<\/h3>\n
\"\"<\/a>
JSON output<\/figcaption><\/figure>\n
\n
Try it!<\/a><\/div>\n<\/div>\n
Search Syntax<\/h2>\n
The examples shown may be a good starting point but are by no means the only queries that may be made. You can view all search query syntaxes<\/a> and create your own as needed.<\/p>\n
Conclusion<\/h2>\n
As you can see, with Sourcegraph, the Fedora Linux community can now quickly search for all code hosted at src.fedoraproject.org<\/a>, regardless of whether they are literal or complex regex queries.<\/p>\n
I appreciate the Fedora Linux community being so helpful and welcoming. If you have anything you want to add or questions, my team and I will be in the comments section below. You can also join us on Slack<\/a>.<\/p>\n
Special thanks to Vanesa Ortiz<\/a> for making this collaboration happen<\/a>, Ben Venker<\/a> for his help fixing my broken regex (multiple times), as well as Rebecca Dodd<\/a> and Nick Moore<\/a> for their help with editing.<\/p>\n","protected":false},"excerpt":{"rendered":"
In October 2021, a Fedora Linux user asked a question about licensing. Fedora Project Leader Matthew Miller left a response: \u201cSince we don’t have a complete, exploded, searchable repository of all of the packages in Fedora, I don’t have a quick way to check.\u201d  Followed by: \u201c…or possibly pay Sourcegraph to do it for us. […]<\/p>\n","protected":false},"author":2,"featured_media":123524,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[48],"tags":[45,61,46,47,77],"class_list":["post-123523","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-fedora-os","tag-fedora","tag-fedora-project-community","tag-magazine","tag-news","tag-software"],"_links":{"self":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/123523","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/comments?post=123523"}],"version-history":[{"count":0,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/123523\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media\/123524"}],"wp:attachment":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media?parent=123523"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/categories?post=123523"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/tags?post=123523"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}