Microsoft

Microsoft is Busy Rewriting Core Windows Code in Memory-safe Rust (theregister.com) 150

Microsoft is rewriting core Windows libraries in the Rust programming language, and the more memory-safe code is already reaching developers. From a report: David "dwizzle" Weston, director of OS security for Windows, announced the arrival of Rust in the operating system's kernel at BlueHat IL 2023 in Tel Aviv, Israel, last month. "You will actually have Windows booting with Rust in the kernel in probably the next several weeks or months, which is really cool," he said. "The basic goal here was to convert some of these internal C++ data types into their Rust equivalents."

Microsoft showed interest in Rust several years ago as a way to catch and squash memory safety bugs before the code lands in the hands of users; these kinds of bugs were at the heart of about 70 percent of the CVE-listed security vulnerabilities patched by the Windows maker in its own products since 2006. The Rust toolchain strives to prevent code from being built and shipped that is exploitable, which in an ideal world reduces opportunities for miscreants to attack weaknesses in software. Simply put, Rust is focused on memory safety and similar protections, which cuts down on the number of bad bugs in the resulting code. Rivals like Google have already publicly declared their affinity for Rust.

Open Source

Linux Kernel 6.3 Released (zdnet.com) 16

An anonymous reader quotes a report from ZDNet, written by Steven Vaughan-Nichols: The latest Linux kernel is out with a slew of new features -- and, for once, this release has been nice and easy. [...] Speaking of Rust, everyone's favorite memory-safe language, the new kernel comes with user-mode Linux support for Rust code. Miguel Ojeda, the Linux kernel developer, who's led the efforts to bring Rust to Linux, said the additions mean we're, "getting closer to a point where the first Rust modules can be upstreamed."

Other features in the Linux 6.3 kernel include support and enablement for upcoming and yet-to-be-released Intel and AMD CPUs and graphics hardware. While these updates will primarily benefit future hardware, several changes in this release directly impact today's users' day-to-day experience. The kernel now supports AMD's automatic Indirect Branch Restricted Speculation (IBRS) feature for Spectre mitigation, providing a less performance-intensive alternative to the retpoline speculative execution.

Linux 6.3 also includes new power management drivers for ARM and RISC-V architectures. RISC-V has gained support for accelerated string functions via the Zbb bit manipulation extension, while ARM received support for scalable matrix extension 2 instructions. For filesystems, Linux 6.3 brings AES-SHA2-based encryption support for NFS, optimizations for EXT4 direct I/O performance, low-latency decompression for EROFS, and a faster Brtfs file-system driver. Bottom line: many file operations will be a bit more secure and faster.

For gamers, the new kernel provides a native Steam Deck controller interface in HID. It also includes compatibility for the Logitech G923 Xbox edition racing wheel and improvements to the 8BitDo Pro 2 wired game controllers. Who says you can't game on Linux? Single-board computers, such as BannaPi R3, BPI-M2 Pro, and Orange Pi R1 Plus, also benefit from updated drivers in this release. There's also support for more Wi-Fi adapters and chipsets. These include: Realtek RTL8188EU Wi-Fi adapter support; Qualcomm Wi-Fi 7 wireless chipset support; and Ethernet support for NVIDIA BlueField 3 DPU. For users dealing with complex networks that have both old-school and modern networks, the new kernel can also handle multi-path TCP handling mixed flows with IPv4 and IPv6.
Linux 6.3 is available from kernel.org. You can learn how to compile the Linux kernel yourself here.
Programming

New Version of Rust Speeds Compilation With Less Debugging Info By Default (phoronix.com) 24

The Rust team released a new version Thursday — Rust 1.69.0 — boasting over over 3,000 new commits from over 500 contributors.

Phoronix highlights two new improvements: In order to speed-up compilation speeds, Rust 1.69 and moving forward debug information is no longer included in build scripts by default. Cargo will avoid emitting debug information in build scripts by default — leading to less informative backtraces in build scripts when problems arise, but faster build speeds by default. Those wanting the debug information emitted can now set the debug flag in their Cargo.toml configuration.

The Cargo build shipped by Rust 1.69 is also now capable of suggesting fixes automatically for some of the generated warnings. Cargo will also suggest using "cargo fix" / "cargo clippy --fix" when it knows the errors can be automatically fixed.

Programming

Rust Foundation Apologizes For Proposed Trademark Changes, Promises Improvement (theregister.com) 37

"The Rust Foundation on Monday apologized for confusion caused by the organization's proposed trademark policy changes," reports the Register.

The Foundation now says their proposed policy "clearly needs improvement" and "there are many valid critiques of the initial draft," promising to address them and adopt a more transparent process (with a report summarizing the feedback soon). From the Register's report: The foundation, which provides financial and legal support for the memory-safe programming language, had proposed fresh rules on the use of the word Rust and its logo, which included the recommendation that people not use 'Rust' in their Rust crate names, eg: vulture-rs would be preferred over vulture-rust. These draft changes triggered a backlash... Over the weekend, Rust creator Graydon Hoare voiced support for the community's objections in a Reddit discussion thread, in response to a post by programmer Andrew Gallant, a former member of the Rust moderation team, who argued the new policy was not all that different from the old one.

"Open them up side by side — old and new — and look at what they each say about, specifically, package names, project names, repos or websites using the word 'rust', or modified versions of the logo used for small groups or projects," wrote Hoare. "These are specifically the things people are upset about, because they all changed from 'acceptable' to 'prohibited' when 'clarifying' the policy. And those are specifically things that everyone in the community does, and has done, for years. There are zillions of packages, projects, repos, websites and groups using the names and logo this way, as the old policy said they could. The new policy tells them all to stop."

Long-time open source advocate Bruce Perens told the Register that Rust's trademark policy "goes far awry of fair use which is legally permitted. Books on Rust will always have its name in their title, commercial products will be advertised as being written in Rust, being compatible with Rust, or compiling Rust. But the policy attempts to deny permission for these things. A proper trademark policy prevents others from representing that their product is Rust or is endorsed by the trademark holder of Rust. That's really as much as you can ever enforce, so there's no sense in a policy that asks for more."

The Register also spoke to Ashley Williams, a former member of the Rust core team and the original executive director and founder of the Rust Foundation, who argued upheaval in Rust's governance over the past year led to a team with less experience dealing with the Rust community. "I think a couple of very passionate people participated in the trademark working group and they didn't involve a lot of people who have even basic experience interacting with the community. So really classic community behaviors ended up getting prohibited in that [draft] policy. And that's really why everybody got upset. The policy ultimately said, 'a thing that you do all the time as a way of contributing to the Rust community is now against our policy.'"
Programming

Undercutting Microsoft, Amazon Offers Free Access to Its AI Coding Assistant 'CodeWhisperer' (theverge.com) 45

Amazon is making its AI-powered coding assistant CodeWhisperer free for individual developers, reports the Verge, "undercutting the $10 per month pricing of its Microsoft-made rival." Amazon launched CodeWhisperer as a preview last year, which developers can use within various integrated development environments (IDEs), like Visual Studio Code, to generate lines of code based on a text-based prompt....

CodeWhisperer automatically filters out any code suggestions that are potentially biased or unfair and flags any code that's similar to open-source training data. It also comes with security scanning features that can identify vulnerabilities within a developer's code, while providing suggestions to help close any security gaps it uncovers. CodeWhisperer now supports several languages, including Python, Java, JavaScript, TypeScript, and C#, including Go, Rust, PHP, Ruby, Kotlin, C, C++, Shell scripting, SQL, and Scala.

Here's how Amazon's senior developer advocate pitched the usefulness of their "real-time AI coding companion": Helping to keep developers in their flow is increasingly important as, facing increasing time pressure to get their work done, developers are often forced to break that flow to turn to an internet search, sites such as StackOverflow, or their colleagues for help in completing tasks. While this can help them obtain the starter code they need, it's disruptive as they've had to leave their IDE environment to search or ask questions in a forum or find and ask a colleague — further adding to the disruption. Instead, CodeWhisperer meets developers where they are most productive, providing recommendations in real time as they write code or comments in their IDE. During the preview we ran a productivity challenge, and participants who used CodeWhisperer were 27% more likely to complete tasks successfully and did so an average of 57% faster than those who didn't use CodeWhisperer....

It provides additional data for suggestions — for example, the repository URL and license — when code similar to training data is generated, helping lower the risk of using the code and enabling developers to reuse it with confidence.

Programming

Rust Foundation Solicits Feedback on Updated Policy for Trademarks (google.com) 41

"Rust" and "Cargo" are registered trademarks held by the Rust Foundation — the independent non-profit supporting Rust's maintainers. In August 1,000 people responded to the foundation's Trademark Policy Review Survey, after which the foundation invited any interested individuals to join their Trademark Policy Working Group (which also included Rust Project leaders). They've now created a draft of an updated policy for feedback...

Crate, RS, "Rustacean," and the logo of Ferris the crab are all available for use by anyone consistent with their definition, with no special permission required. Here's how the document's quick reference describes other common use-cases:
  • Selling Goods — Unless explicitly approved, use of the Rust name or Logo is not allowed for the purposes of selling products/promotional goods for gain/profit, or for registering domain names. For example, it is not permitted to sell stickers of the Rust logo in an online shop for your personal profit.
  • Showing Support of Rust — When showing your support of the Rust Project on a personal site or blog, you may use the Rust name or Logo, as long as you abide by all the requirements listed in the Policy. You may use the Rust name or Logo in social media handles, avatars, and emojis to demonstrate Rust Project support in a manner that is decorative, so long as you don't suggest commercial Rust affiliation.
  • Inclusion of the Marks in Educational Materials — You may use the Rust name in book and article titles and the Logo in graphic components, so long as you make it clear that the Rust Project or Foundation has not reviewed/approved/endorsed your content.

There's also a FAQ, answering questions like "Can I use the Rust logo as my Twitter Avatar?" The updated policy draft says "We consider social media avatars on personal accounts to be fair use. On the other hand, using Rust trademarks in corporate social media bios/profile pictures is prohibited.... In general, we prohibit the modification of the Rust logo for any purpose, except to scale it. This includes distortion, transparency, color-changes affiliated with for-profit brands or political ideologies. On the other hand, if you would like to change the colors of the Rust logo to communicate allegiance with a community movement, we simply ask that you run the proposed logo change by us..."

And for swag at events using the Rust logo, "Merch developed for freebies/giveaways is normally fine, however you need approval to use the Rust Word and/or Logo to run a for-profit event. You are free to use Ferris the crab without permission... If your event is for-profit, you will need approval to use the Rust name or Logo. If you are simply covering costs and the event is non-profit, you may use the Rust name or Logo as long as it is clear that the event is not endorsed by the Rust Foundation. You are free to use Ferris the crab without permission."


Programming

C Rival 'Zig' Cracks Tiobe Index Top 50, Go Remains in Top 10 (infoworld.com) 167

InfoWorld reports: Zig, a general purpose programming language that interacts with C/C++ programs and promises to be a modern alternative to C, has made an appearance in the Tiobe index of programming language popularity. Zig entered the top 50 in the April edition of the Tiobe Programming Community Index, ranking 46th, albeit with a rating of just 0.19%. By contrast, the Google-promoted Carbon language, positioned as an experimental successor to C++, ranked just 168th.
Tiobe CEO Paul Jansen argues that high-performance languages "are booming due to the vast amounts of data that needs to be processed nowadays. As a result, C and C++ are doing well in the top 10 and Rust seems to be a keeper in the top 20." Zig has all the nice features of C and C++ (such as explicit memory management enhanced with option types) and has abandoned the not-so-nice features (such as the dreadful preprocessing). Entering the top 50 is no guarantee to become a success, but it is at least a first noteworthy step. Good luck Zig!
Tiobe bases its monthly ranking of programming language popularity on search engine results for courses, third party vendors, and engineers. Here's what they's calculated for the most popular programming languages in April of 2023:
  • Python
  • C
  • Java
  • C++
  • C#
  • Visual Basic
  • JavaScript
  • SQL
  • PHP
  • Go

April's top 10 was nearly identical to the rankings a year ago, but assembly language fell from 2022's #8 position to #12 in 2023. SQL and PHP rose one rank (into 2023's #8 and #9 positions) — and as in March, the rankings now shows Go as the 10th most popular programming language.


Programming

Collabora Developer Explores Rust Support for the Linux Kernel's V4L2/Media Subsystem (phoronix.com) 5

On Thursday patches were submitted for bringing Rust infrastructure to the Video 4 Linux 2 (V4L2) framework (within Linux's media subsystem) by Collabora's Daniel Almeida.

Phoronix reports: This provides just enough for working with a prototype VirtIO camera driver written in Rust along with a Rust sample driver. These initial patches are just intended to start the discussion around V4L2 Rust driver support and the actual upstreaming of the Rust support for these camera drivers may still be some ways down the line.
Programming

Meet Zig: the Modern Alternative to the C Programming Language (infoworld.com) 117

Systems-oriented developers already have programming languages like C, C++, Rust, and Go, notes InfoWorld.

But now, "we also have Zig, a newer language that seeks to absorb what's best about these languages and offer comparable performance with a better, more reliable developer experience." Zig is a very active project. It was started by Andrew Kelley in 2015 and now seems to be reaching critical mass. Zig's ambition is rather momentous in software history: to become the heir to C's longstanding reign as both the go-to portable low-level language and as a standard to which other languages are compared....

Currently, Zig is being used to implement the Bun.js JavaScript runtime as an alternative to Node.js. Bun's creator Jarred Sumner told me "Zig is sort of similar to writing C, but with better memory safety features in debug mode and modern features like defer (sort of similar to Go's) and arbitrary code can be executed at compile-time via comptime. It has very few keywords so it's a lot easier to learn than C++ or Rust."

Zig differs from most other languages in its small feature footprint, which is the outcome of an explicit design goal: Only one obvious way to do things. Zig's developers take this goal so much to heart that for a time, Zig had no for loop, which was deemed an unnecessary syntactic elaboration upon the already adequate while loop. Kevin Lynagh, coming from a Rust background, wrote, "The language is so small and consistent that after a few hours of study I was able to load enough of it into my head to just do my work." Nathan Craddock, a C developer, echoed the sentiment. Programmers seem to really like the focused quality of Zig's syntax.

While Zig is "approaching" production-ready status, the article notes its high degree of interoperability with C and C++, its unique error-handling system, and its elimination of a malloc keyword (leaving memory allocation to the standard library).

"For now, the Zig team appears to be taking its time with the 1.0 release, which may drop in 2025 or later — but none of that stops us from building all sorts of things with the language today."
Programming

Rust Project Reveals New 'Constitution' in Wake of Crisis (thenewstack.io) 81

"The Rust open source project, which handles standards for the language, released a new governance plan Thursday," reports The New Stack, "the cumulation of six months of intense work." Released as a request for comment on GitHub, it will now undergo a comment period. It requires ratification by team leaders before it's accepted.

The Rust project interacts with, but is separate from, the Rust Foundation, which primarily handles the financial assets of Rust. Two years ago, the project had a very public blowup after its entire mod team resigned and publicly posted a scathing account of the core team, which the mod team called "unaccountable to anyone but themselves." It even suggested the core team was not to be trusted, although the team later recanted and apologized for that.

[Rust core team developer] Josh Triplett understandably didn't want to dwell on the kerfuffle that lead to this action. He focused instead on the underlying structural issues that lead to the leadership crisis. "As a result of that, there was widespread agreement within the project that we needed to create a better formal governance structure that removed some of those ambiguities and conflicts, and had mechanisms for dealing with this without ever having a similar crisis," Triplett told The New Stack. "We don't want to ever to have things get to that point again...."

The original Rust project governance structure evolved out of Mozilla, where Rust began and was nurtured for years. Around 2016 or 2017, a request for comment came out that established the Rust project's governance, Triplett said. It created approximately six teams, including the core, language, mod, library and cargo teams. Among the problems with the old model was that the core team became responsible for not just overseeing problems that arose, but solving them as well, Triplett said. That led to burnout and problems, said JT Turner, one of the co-authors on the new model and a member of the Rust core team.... Ultimately, the old governance model was "not a very precise document," Triplett added.

"It was just, 'Hey, here's the rough divisions of power,' and because that document was very rough and informal, it didn't scale to today," he said. "That's one of the things that led to the governance crisis."

Programming

GCC Gets a New Frontend for Rust (fosdem.org) 106

Slashdot reader sleeping cat shares a recent FOSDEM talk by a compiler engineer on the team building Rust-GCC, "an alternative compiler implementation for the Rust programming language."

"If gccrs interprets a program differently from rustc, this is considered a bug," explains the project's FAQ on GitHub.

The FAQ also notes that LLVM's set of compiler technologies — which Rust uses — "is missing some backends that GCC supports, so a gccrs implementation can fill in the gaps for use in embedded development." But the FAQ also highlights another potential benefit: With the recent announcement of Rust being allowed into the Linux Kernel codebase, an interesting security implication has been highlighted by Open Source Security, inc. When code is compiled and uses Link Time Optimization (LTO), GCC emits GIMPLE [an intermediate representation] directly into a section of each object file, and LLVM does something similar with its own bytecode. If mixing rustc-compiled code and GCC-built code in the Linux kernel, the compilers will be unable to perform a full link-time optimization pass over all of the compiled code, leading to absent CFI (control flow integrity).

If Rust is available in the GNU toolchain, releases can be built on the Linux kernel (for example) with CFI using LLVM or GCC.

Started in 2014 (and revived in 2019), "The effort has been ongoing since 2020...and we've done a lot of effort and a lot of progress," compiler engineer Arthur Cohen says in the talk. "We have upstreamed the first version of gccrs within GCC. So next time when you install GCC 13 — you'll have gccrs in it. You can use it, you can start hacking on it, you can please report issues when it inevitably crashes and dies horribly."

"One big thing we're doing is some work towards running the rustc test suite. Because we want gccrs to be an actual Rust compiler and not a toy project or something that compiles a language that looks like Rust but isn't Rust, we're trying really hard to get that test suite working."

Read on for some notes from the talk...
Programming

How Rust Went From a Side Project To the World's Most-Loved Programming Language (technologyreview.com) 118

An anonymous reader quotes a report from MIT Technology Review: Many software projects emerge because -- somewhere out there -- a programmer had a personal problem to solve. That's more or less what happened to Graydon Hoare. In 2006, Hoare was a 29-year-old computer programmer working for Mozilla, the open-source browser company. Returning home to his apartment in Vancouver, he found that the elevator was out of order; its software had crashed. This wasn't the first time it had happened, either. Hoare lived on the 21st floor, and as he climbed the stairs, he got annoyed. "It's ridiculous," he thought, "that we computer people couldn't even make an elevator that works without crashing!" Many such crashes, Hoare knew, are due to problems with how a program uses memory. The software inside devices like elevators is often written in languages like C++ or C, which are famous for allowing programmers to write code that runs very quickly and is quite compact. The problem is those languages also make it easy to accidentally introduce memory bugs -- errors that will cause a crash. Microsoft estimates that 70% of the vulnerabilities in its code are due to memory errors from code written in these languages.

Most of us, if we found ourselves trudging up 21 flights of stairs, would just get pissed off and leave it there. But Hoare decided to do something about it. He opened his laptop and began designing a new computer language, one that he hoped would make it possible to write small, fast code without memory bugs. He named it Rust, after a group of remarkably hardy fungi that are, he says, "over-engineered for survival." Seventeen years later, Rust has become one of the hottest new languages on the planet -- maybe the hottest. There are 2.8 million coders writing in Rust, and companies from Microsoft to Amazon regard it as key to their future. The chat platform Discord used Rust to speed up its system, Dropbox uses it to sync files to your computer, and Cloudflare uses it to process more than 20% of all internet traffic.

When the coder discussion board Stack Overflow conducts its annual poll of developers around the world, Rust has been rated the most "loved" programming language for seven years running. Even the US government is avidly promoting software in Rust as a way to make its processes more secure. The language has become, like many successful open-source projects, a barn-raising: there are now hundreds of die-hard contributors, many of them volunteers. Hoare himself stepped aside from the project in 2013, happy to turn it over to those other engineers, including a core team at Mozilla. It isn't unusual for someone to make a new computer language. Plenty of coders create little ones as side projects all the time. But it's meteor-strike rare for one to take hold and become part of the pantheon of well-known languages alongside, say, JavaScript or Python or Java. How did Rust do it?

Programming

Can C++ Be Safer? Bjarne Stroustrup On Ensuring Memory Safety (thenewstack.io) 110

C++ creator Bjarne Stroustrup "joins calls for changing the programming language itself to address security concerns," according to an article shared by Slashdot user guest reader: In mid-January, the official C++ "direction group" -- which makes recommendations for the programming language's evolution -- issued a statement addressing concerns about C++ safety. While many languages now support "basic type safety" -- that is, ensuring that variables access only sections of memory that are clearly defined by their data types -- C++ has struggled to offer similar guarantees.

This new statement, co-authored by C++ creator Bjarne Stroustrup, now appears to call for changing the C++ programming language itself to address safety concerns. "We now support the idea that the changes for safety need to be not just in tooling, but visible in the language/compiler, and library." The group still also supports its long-preferred use of debugging tools to ensure safety (and "pushing tooling to enable more global analysis in identifying hard for humans to identify safety concerns"). But that January statement emphasizes its recommendation for changes within C++.

Specifically, it proposes "packaging several features into profiles" (with profiles defined later as "a collection of restrictions and requirements that defines a property to be enforced" by, for example, triggering an automatic analysis.) In this way the new changes for safety "should be visible such that the Safe code section can be named (possibly using profiles), and can mix with normal code." And this new approach would ultimately bring not just safety but also flexibility, with profiles specifically designed to support embedded computing, performance-sensitive applications, or highly specific problem domains, like automotive, aerospace, avionics, nuclear, or medical applications.

"For example, we might even have safety profiles for safe-embedded, safe-automotive, safe-medical, performance-games, performance-HPC, and EU-government-regulation," the group suggests. Elsewhere in the document they put it more succinctly. "To support more than one notion of 'safety', we need to be able to name them."

Stroustrup emphasized his faith in C++ in a 2020 interview. "I think C++ can do anything Rust can do, and I would like it to be much simpler to use," Stroustrup told the Association for Computing Machinery's Special Interest Group on Programming Languages.

But even then, he'd said that basic type safety was one of his earliest design goals -- and one he's spent decades trying to achieve. "I get a little bit sad when I hear people talk about C++ as if they were back in the 1980s, the 1990s, which a lot of people do. They looked at it back in the dark ages, and they haven't looked since."
Programming

A Developer is Reimplementing GNU's Core Utilities in Rust (phoronix.com) 186

A Rust-based re-implementation of GNU core utilities like cp and mv is "reaching closer to parity with the widely-used GNU upstream and becoming capable of taking on more real-world uses," reports Phoronix: Debian developer Sylvestre Ledru [also an engineering director at Mozilla] began working on uutils during the COVID-19 pandemic and presented last week at FOSDEM 2023 on his Coreutils replacement effort. With uutils growing into increasingly good shape, it's been packaged up by many Linux distributions and is also used now by "a famous social network via the Yocto project...."

The goals with uutils are to try to create a drop-in replacement for GNU Coreutils, strive for good cross-platform support, and easy testing. Ledru's initial goals were about being able to boot Debian, running the most popular packages, building key open-source software, and all-around it's been panning out to be a great success.... [M]ore performance optimizations are to come along with other work for compatibility against the GNU tools and implementing some still missing options in different programs

Portables

System76 Announces Redesigned 'Pangolin' AMD/Linux Laptop (9to5linux.com) 42

System76 is announcing a "fully redesigned" version of its AMD-only Linux-powered "Pangolin" laptop with an upgraded memory, storage, processor, and display.

9to5Linux reports: It features the AMD Ryzen 7 6800U processor with up to 4.7 GHz clock speeds, 8 cores, 16 threads, and AMD Radeon 680M integrated graphics.... a 15.6-inch 144Hz Full HD (1920 x 1080) display [using 12 integrated Radeon graphics cores] with a matte finish, a sleek magnesium alloy chassis, and promises up to 10 hours of battery life with its 70 Wh Li-Ion battery. It also features a single-color backlit US QWERTY Keyboard and a multitouch clickpad. Under the hood, the Linux-powered laptop boasts 32 GB LPDDR5 6400 MHz of RAM and it can be equipped with up to 16TB PCIe 4.0 NVMe M.2 SSD storage. Another cool feature is the hardware camera kill switch for extra privacy....

As with all of System76's Linux-powered laptops, the all-new Pangolin comes pre-installed with System76's in-house built Pop!_OS Linux distribution featuring the GNOME-based COSMIC desktop and full disk-encryption or with Ubuntu 22.04 LTS.

Programming

GitHub Claims Source Code Search Engine Is a Game Changer (theregister.com) 39

Thomas Claburn writes via The Register: GitHub has a lot of code to search -- more than 200 million repositories -- and says last November's beta version of a search engine optimized for source code that has caused a "flurry of innovation." GitHub engineer Timothy Clem explained that the company has had problems getting existing technology to work well. "The truth is from Solr to Elasticsearch, we haven't had a lot of luck using general text search products to power code search," he said in a GitHub Universe video presentation. "The user experience is poor. It's very, very expensive to host and it's slow to index." In a blog post on Monday, Clem delved into the technology used to scour just a quarter of those repos, a code search engine built in Rust called Blackbird.

Blackbird currently provides access to almost 45 million GitHub repositories, which together amount to 115TB of code and 15.5 billion documents. Shifting through that many lines of code requires something stronger than grep, a common command line tool on Unix-like systems for searching through text data. Using ripgrep on an 8-core Intel CPU to run an exhaustive regular expression query on a 13GB file in memory, Clem explained, takes about 2.769 seconds, or 0.6GB/sec/core. [...] At 0.01 queries per second, grep was not an option. So GitHub front-loaded much of the work into precomputed search indices. These are essentially maps of key-value pairs. This approach makes it less computationally demanding to search for document characteristics like the programming language or word sequences by using a numeric key rather than a text string. Even so, these indices are too large to fit in memory, so GitHub built iterators for each index it needed to access. According to Clem, these lazily return sorted document IDs that represent the rank of the associated document and meet the query criteria.

To keep the search index manageable, GitHub relies on sharding -- breaking the data up into multiple pieces using Git's content addressable hashing scheme and on delta encoding -- storing data differences (deltas) to reduce the data and metadata to be crawled. This works well because GitHub has a lot of redundant data (e.g. forks) -- its 115TB of data can be boiled down to 25TB through deduplication data-shaving techniques. The resulting system works much faster than grep -- 640 queries per second compared to 0.01 queries per second. And indexing occurs at a rate of about 120,000 documents per second, so processing 15.5 billion documents takes about 36 hours, or 18 for re-indexing since delta (change) indexing reduces the number of documents to be crawled.

Stats

Can Stack Overflow's Survey Predict Next Year's Most Loved Programming Language? (stackoverflow.blog) 46

What happens when Stack Overflow's senior research analyst delves more deeply into results from their annual Developer Survey? Rust, Elixir, Clojure, Typescript, and Julia are at the top of the list of Most Loved Programming Languages. However, in looking at the last three years, we see a bit of movement. [While Rust has remained #1 since 2020, Elixir has risen to #2, while Clojure and TypeScript have dropped.]

In 2022, we added a drill-down to specifically show popularity amongst those learning to code. Because Stack Overflow is a learning resource, I would expect that popularity amongst those specifically learning would be a good indicator of current and future programming language popularity. There is an interesting pattern in comparing Most Loved and Learning to Code Popularity: people learning to code aren't using the most loved languages....


Less than 1% of those learning responded they were using either Clojure or Elixir.

1.2% are using Julia
7.1% are using Rust
and 15.1% are using Typescript.

The article still tries to tease out ways to predict future popular programming languages (by, for example, the number of questions being asked about languages, especially by new programmers learning to code). But along the way, they uncover other surprising statistical truths about the limits of their data:
  • "Stack Overflow questions are more susceptible to the preferences of those using the site as a learning tool rather than those of more advanced developers."
  • "[B]eing loved (via the Developer Survey) is not related to generating more questions on Stack Overflow. And this makes sense: posting questions most likely speaks to friction with coding, a friction that may lead to loving a programming language less."
  • "Our latest Developer Survey showed us that ~32% of programmers have been professionally coding for four years or less, a significant amount of people who are most likely involved in learning programming languages. That is, beginner-friendly languages get the most questions and popularity, but the Most Loved languages make veteran developers happy."

Ubuntu

Canonical Announces General Availability of Ubuntu Pro, Free for Up to 5 PCs (9to5linux.com) 52

An anonymous reader quotes a report from 9to5Linux: Ubuntu maker Canonical announced Thursday the general availability of its Ubuntu Pro comprehensive subscription for Ubuntu users who want to expand the security updates and compliance of their systems.

First released in a beta version in October 2022 with free subscriptions for personal and small-scale commercial use on up to 5 machines, Ubuntu Pro is only available for Ubuntu LTS (Long-Term Support) releases, starting with Ubuntu 16.04, and promises up to 10 years of security updates, as well as access to exclusive tools. These include Ansible, Apache Tomcat, Apache Zookeeper, Docker, Drupal, Nagios, Node.js, phpMyAdmin, Puppet, PowerDNS, Python 2, Redis, Rust, WordPress, ROS, and many others.

The Ubuntu Pro subscription promises patches for critical CVEs in less than 24 hours and expands the optional technical support to an additional 23,000 open-source packages and toolchains beyond the main operating system, not just for Ubuntu's main software repository....

Canonical says that if you need Ubuntu Pro for more than five PCs, you will have to purchase a paid plan, which is currently priced at $25 USD per year for workstations or $500 USD per year for servers with a 30-day free trial. Official Ubuntu Community members get free support for up to 50 machines.

Programming

Rust Safety Is Not Superior To C++, Bjarne Stroustrup Says (open-std.org) 220

guest reader writes: The Open Standards site contains a new paper from Bjarne Stroustrup titled A call to action: Think seriously about "safety"; then do something sensible about it.

Bjarne reacts to an NSA report about Software Memory Safety since the report excludes C and C++ as unsafe. Bjarne does not consider any of the report's choices for "safe" languages as superior to C++ for the range of uses he cares about.

From Bjarne's response: I have worked for decades to make it possible to write better, safer, and more efficient C++. In particular, the work on the C++ Core Guidelines specifically aims at delivering statically guaranteed type-safe and resource-safe C++ for people who need that without disrupting code bases that can manage without such strong guarantees or introducing additional tool chains. For example, the Microsoft Visual Studio analyzer and its memory-safety profile deliver much of the CG support today and any good static analyzer (e.g., Clang tidy, that has some CG support) could be made to completely deliver those guarantees at a fraction of the cost of a change to a variety of novel "safe" languages.
Bjarne also complains that in the NSA's document, "'safe' is limited to memory safety, leaving out on the order of a dozen other ways that a language could (and will) be used to violate some form of safety and security." There is not just one definition of "safety", and we can achieve a variety of kinds of safety through a combination of programming styles, support libraries, and enforcement through static analysis.... I envision compiler options and code annotations for requesting rules to be enforced. The most obvious would be to request guaranteed full type-and-resource safety.
Bjarne notes that if you work in application domains which prioritize performance over type safety, you could "apply the safety guarantees only where required and use your favorite tuning techniques where needed." Partial adoption of some of the rules (e.g., rules for range checking and initialization) is likely to be important. Gradual adoption of safety rules and adoption of differing safety rules will be important. If for no other reason than the billions of lines of C++ code will not magically disappear, and even "safe" code (in any language) will have to call traditional C or C++ code or be called by traditional code that does not offer specific safety guarantees.

Ignoring the safety issues would hurt large sections of the C++ community and undermine much of the other work we are doing to improve C++.

The article also contains the following references for consideration:
- Design Alternatives for Type-and-Resource Safe C++.
- Type-and-resource safety in modern C++.
- A brief introduction to C++'s model for type- and resource-safety.
- C++ Core Guidelines, safety profiles.
Chromium

Google To Allow Rust Code In the Chromium Browser (phoronix.com) 23

Google announced today that moving forward they will be allowing Rust code into the Chromium code-base, the open-source project that ultimately served as the basis for their Chrome web browser. Phoronix reports: Google is working to introduce a production Rust toolchain into their build system for Chromium and will be allowing Rust libraries for use within Chrome/Chromium. The timeframe for getting this all together is expected within the next year following a slow ramp. Google is backing Rust for Chromium to allow for simpler and safer code than "complex C++" overall, particularly around avoiding memory safety bugs. In turn using Rust should help speed-up development and improve overall security of the Chrome web browser. Initially they are focused on supporting interop in a single direction from C++ to Rust and for now will only be supporting third-party libraries for their Rust usage.

Slashdot Top Deals