Skip to main content
Category

News

Sustaining long-term support for Linux mainline based products

By News

We all know that introducing and supporting hardware in Linux mainline is challenging. At KernelCI, we want to understand how we can help move the ecosystem further upstream. 

The challenges come from many angles:

  • Invest in upstreaming and maintain the hardware support into Linux (and other projects) in the long-run
  • Take IP protection in consideration, making sure that no critical information is shared in the open
  • Time to market pressures for the product release
  • Test mainline based products (continuously), making sure:
    • The hardware remains functional
    • No regressions appear
  • Maintain test infrastructure, so testing can be timely and efficient
  • Interact with the kernel community to improve the state of the art and fix issues
  • And so much more

These challenges come from both technical and business aspects. On the one hand, there are still a lot of difficulties in interacting with the upstream community to implement drivers, review patches, and solve regressions. On the other hand, business stakeholders and decision-makers have a hard time understanding upstream practices, which contributes to insufficient investment to bring enough knowledge and resources for effective participation in upstream. KernelCI aims to help Hardware Vendors address both the technical and business obstacles.

With that in mind, KernelCI wants to start a discussion with all interested Hardware Vendors to share our experiences and pain points. We could then look ahead at how we can collaborate on improving the kernel integration processes and facilitate the maintenance of stable kernels in the long run. Stability and security needs are growing exponentially, with so much of our global infrastructure depending on the hardware and software we build.

If you are in Vienna, Austria on September 18th for Open Source Summit Europe and/or Linux Plumbers Conference, we invite you to join our in-person discussion to happen from 1:30pm to 3pm at LPC Room 1.34.
If you have any questions or comments, please contact us at kernelci-members@groups.io.

KernelCI at Linux Plumbers Conference 2024

By Blog, News

The KernelCI community will have a busy few days at the upcoming Linux Plumbers Conference(LPC) in Vienna, Austria. We will be presenting 2 talks in the Refereed track, 4 Micro-conferences sessions and 1 BoF.

For those interested in an overview of everything happening in KernelCI, come to the “Meeting the New KernelCI” presentation from Don Zickus (Red Hat) and Gustavo Padovan(Collabora) in LPC’s Refereed Track Thursday at 12:00pm CEST.

Following the talk, on Thursday afternoon, starting at 3pm, the KernelCI team will set up a small corner in the “hallway track” to dive into more discussions with the community. Come talk to us and dive further in usecases, demo the dashboard, collect feedback, etc.

Hardware Vendors are key for KernelCI success, so we will be hosting a discussion with Hardware Vendors to understand how we can improve the kernel testing and integration overall and how KernelCI, as an organization can help drive progress. This will happen Wednesday at 1:30pm to 3:00pm (Room 1.34). If you are interested in joining please send an email to kernelci-members@groups.io. Open Source Summit attendees are also welcome in this session.

Apart from that we will also bring several technical challenges for discussions in the micro-conferences, host a talk on GitLab and a Kernel Testing BoF. Check our full agenda:

Wednesday

Thursday

  • 12:00 Meet the New KernelCI in the LPC Refereed Track (Hall L2/L3). By Don Zickus (Red Hat), Gustavo Padovan(Collabora)
  • 15:00 – 18:00 KernelCI Booth in the Hallway Track. By the KernelCI community.
  • 19:00 – 21:00 KernelCI Happy Hour – Location: The View Café-Bar-Restaurant. 25 min walk from the conference venue.

Friday

Times are in the Vienna/CEST Timezone.

As you can see it is a pretty busy agenda. If you want to talk to us please reach out at  kernelci-members@groups.io. We will be happy to see you at LPC!

Welcoming Texas Instruments to KernelCI

By News

We are pleased to announce that Texas Instruments is joining KernelCI as a Premier member. Texas Instruments(TI) is a lead semiconductor company and a long time contributor to the upstream Linux kernel. To better serve its customers, TI has embraced Open Source, thriving to offer upstream support for the chips they produce.

TI is pushing the boundaries of upstream SoC support and advocating for an industry-wide Upstream-First approach. For the past several years, TI has invested in its testing infrastructure and technologies and applied those improvements to TI’s upstream testing. Now, TI is proud to join KernelCI to bring that knowledge, experience, and capability to the larger community, including other semiconductor companies, in order to collaborate openly and improve upstream SoC quality and support.

As Nishanth Menon, a principal open-source technologist at Texas Instruments, says: “KernelCI is crucial for ensuring the longevity of the Linux ecosystem and vital for providing high-quality software in the upstream Linux community, fostering the confidence required for products to be agnostic of vendor software ecosystems.”

The KernelCI board is glad to have TI joining forces to make the new KernelCI testing capabilities stronger. “We are excited to welcome Texas Instruments to the board and looking forward to leveraging their testing capabilities and expertise around SoCs to contribute to our common goal of ensuring the quality, stability and long-term maintenance of the Linux kernel.” said Don Zickus, Chair of the KernelCI Board.

TI is leading the way in creating an open and upstream embedded testing standard for the Linux kernel. All embedded SoC and product vendors incur significant costs to reinvent and maintain the test ecosystem for standard peripherals. KernelCI provides the groundwork to establish a vendor-agnostic testing framework for user experiences at the product level. With this goal in mind, KernelCI is also the perfect umbrella to align the industry and create a better solution by standardizing the test framework, terminology, interfaces, methodology, tools, and reporting.

There couldn’t be a better time for TI to join KernelCI. The project is going through a major overhaul with our new CI architecture coming into place. It replaces our limited legacy system with a quite capable architecture that can leverage different CI systems, hardware and cloud labs, to funnel all the test data to a common place. So, we can analyze, automatic triage and support the regression reporting process together with the Linux kernel community. Our top priority is to serve kernel maintainers and developers to ensure the long-term stability of the Linux kernel.

Strategic updates from the KernelCI project

By Blog, Community, News

The KernelCI project continues on its mission to ensure the quality, stability and long-term maintenance of the Linux kernel. That means supporting the community (especially maintainers) to not just run their code in a Continuous Integration (CI) system, but also deliver relevant, high confidence results and reports. In this post, we will give you a strategic overview of the steps we have taken towards our mission over the past few months.

Enabling new infrastructure to run tests

Our legacy system has shown its age and it has been failing to meet the growing testing needs of the community. It is a quite limited, unstable python2 project that focuses on embedded hardware. We continue on the journey to put in place our new infrastructure, so we can finally move away from the legacy KernelCI system.

We bring good news! The new core service for running tests is already in place, but still going through a stabilization phase. So, the team is ramping up the number of tests slowly to deal with issues that arise especially when it comes to the quality of the testing KernelCI provides. We do not want to repeat past mistakes with results that can’t really be trusted, so our focus right now is on quality rather than quantity. The team is iterating quickly on that process to enable open, wide adoption in the coming months.

In our new KernelCI infrastructure, we already have support to run tests in labs (through LAVA only at the moment), Docker containers, Kubernetes and natively. Adding new labs or test environments should be relatively straightforward. Then, as we add more test environments, we are also laying down the foundation for integration with other CI systems across the community so we can share kernel builds and offer test environments.

As we shared before, the new infrastructure exposes an API for users to create accounts, query results and even drive tests themselves. At the moment, we are focusing on enabling our own pipeline there, so we can shut down the legacy system. But anyone is welcome to request a user account and try it out.

Another initiative from the KernelCI community is the GitLab CI support in the mainline kernel. Here, the goal is to offer maintainers a CI environment that they can manage themselves. With time, KernelCI API can be leveraged to provide a backend for builds and test runs for repositories using GitLab CI.

Trusting tests results and reports

On one end, we are stabilizing our new infrastructure to run the tests. On the other end, we are looking into improving the quality of the reports KernelCI sends out, so maintainers and developers can actually trust them. Given the huge amount of data coming out of test systems and bots today, we must invest in improving the delivery of the results, or else, KernelCI will be contributing to increasing the maintainer burnout rather than helping solve it. That means improving the quality and confidence of the data, so maintainers and developers only receive reports packed with relevant information and no noise or false-positives.

At the time of this writing, we have a handful of trees enabled, boot testing and a few tests enabled (including kselftest support) in our new test infrastructure. That setup is enabling the team to triage ALL the results to identify infrastructure failures and test patterns in general (flakiness, config issue, intermittent issues, etc). There is a significant investment to develop better tests together with the community (like the device tree probe kselftest) that is improving the quality of the results compared to what exists in our legacy system.

As part of the effort, we are developing a layer for post-processing the test results in KCIDB – the KernelCI database to collect test results from the entire Linux kernel test ecosystem. The work in this area is at proof-of-concept level, but it is already enabling the team to evaluate the results coming from our new infrastructure. The post-processing layer should be a key part of the feedback loop with the community. The goal is to increase automation in triaging the results, saving precious time from kernel maintainers. Also, because KCIDB collects data from various CI systems, the post-processing of test results can be enabled for more systems than just KernelCI.

On top of that, the KernelCI team is redesigning the Web Dashboard UX to enable rich visualization of all that data for the entire community. A public request for feedback on UX should go out in the coming weeks.

It’s all about engaging the community in testing

Solving CI needs for the Linux kernel community is not just a technical challenge. It is in big part a community engagement challenge too. The KernelCI project has a strong focus on engaging the community in testing processes. With our new infrastructure coming into place, we are ready to give a new spin to our Community Engagement initiative.

For that, we are forming a Community Engagement Working Group (WG). The WG will focus on connecting with maintainers to discuss and implement improvements in test quality for their subsystems and also act as a feedback recipient for improvements in our post-processing of the test results. The Community Engagement WG will be led by Shuah Khan, kernel maintainer & Linux Fellow at The Linux Foundation.

A dedicated announcement of the Community Engagement WG was sent to the KernelCI mailing list. If you are interested in participating, raise your hand!

Where are we going from here?

As you can see, a lot is going on in KernelCI at the moment. The team is iterating quickly on the development of the new infrastructure, so we will be engaging with new maintainers and developers every month from now on, bringing them to the new infra and pushing the system limit further. If you are a maintainer and want to bring your tests to KernelCI please send us an email at kernelci@lists.linux.dev.

That’s all for now! Stay tuned for updates on topics discussed in this article. Likewise, as the new infrastructure stabilizes, expect a significant amount of documentation updates too.

We thank all KernelCI member organizations and developer community who have been investing in the project over the years. It is only because of the continued support from our community that we are making the legacy system a past story!

Exciting KernelCI TSC updates

By Blog, Community, News

During our last TSC meeting, on March 14th, we had two votes that brought new winds to our community:

Shuah Khan joins the TSC

Shuah, Kernel Maintainer & Linux Fellow at The Linux Foundation, has been elected a new member of the TSC. We believe Shuah’s experience will help the KernelCI project expand its kernel community engagement efforts, taking testing and CI to more parts of the upstream kernel. Welcome, Shuah!

Nikolai Kondrashov elected as the new TSC Chair

Nikolai has been a longstanding contributor under the KernelCI umbrella and the person behind KCIDB. At Red Hat he also contributes to kernel integration technology. His knowledge around the problems KernelCI needs to solve will help the project greatly. His role will be effective on April 1st. All the best to Nikolai in his new role!

While at it, we thank Guillaume for his tenure as TSC Chair of the project!

The detail of the motions which were voted can be found on the TSC documentation.

UX Research for the new Web Dashboard

By Blog, Community, News

Following the UX Analysis RFP we shared a few months ago, we’re now delighted to announce that the KernelCI Advisory Board has accepted a proposal. First of all, we’re grateful to have received such a series of high-quality proposals. Many thanks once again to all the submitters for your interest in the project.

After carefully considering all the options with the board and some TSC members, the vote came out in favour of ProFUSION. Congratulations! We’re looking forward to the next milestones, in fact the work has already started.

Milestones

The UX Analysis project is broken down into three milestones with four weeks assigned to each, so a total of twelve weeks. Some breaks will also be scheduled during this time so the calendar end date is likely to be in March 2024.

M1: Result of interviews

Some interviews will be carried out with various stakeholders to help refine user stories and start modelling the user experience and some design aspects.

M2: First UX design proposal

A first interactive prototype will then be created and made available for members of the public to evaluate and provide feedback.

M3: Final UX design proposal

Based on the feedback and outcome of the first two milestones, the last part is about creating high-level specifications of the web dashboard design suitable for the implementation phase.

Request for Proposals: UX Analysis 2023

By Blog, Community, News

The KernelCI project runs thousands of Linux kernel tests every day, generating a huge amount of data to help the community identify issues and trends. One way to communicate all this is through a bespoke web application that truly embodies the kernel community’s use cases. This Request For Proposals (RFP) aims to be an initial investigation to understand how the User Experience (UX) of the Web Dashboard could look like based on a set of user stories compiled by the KernelCI team.

Budget

We’re expecting vendors to submit a fixed-price proposal showing the total costs for the different phases they plan for the project. Optional or extra phases can be included as well. The vendors have autonomy to propose a process for the iterative feedback roadmap. Payments would be made by the Linux Foundation using the project’s own budget.

Sending proposals

Please take a look at the full RFP-UX-Analysis-2023-v2.pdf document for all the details. Proposals should be sent by email directly to the project members.

The deadline for responding to this RFP is 10th July 2023, six weeks after it has been made public. Then the KernelCI Advisory Board of Members will vote on the 24th July 2023. Exact dates might be subject to change in case of a major practical issue or unavailability of voting members.

Edit: Following a surge in last-minute queries, the timeline has now been extended. The new deadline for submitting proposals is 24th July 2023 and the advisory board is planning to hold a vote on 9th August 2023.

KernelCI at FOSDEM 2023

By Blog, Community, News

Taking place February 4th and 5th, FOSDEM is a fantastic event organised “by the community for the community” in Brussels, Belgium, Europe. FOSDEM provides Open Source communities a place to meet in person.

The KernelCI initiative is delighted to be present at FOSDEM 2023.

Testing is recognized by all to be critically important for Open Source communities at large, and not enough is being done. The KernelCI initiative (a Linux Foundation project) intends to change that.

During the weekend of FOSDEM 2023, please keep testing in mind throughout the conversations you will be having with peers and members of the community.

Remind yourself and your interlocutors of the importance of testing and the existence of the KernelCI initiative.

Find out more about KernelCI

Mission statement: https://kernelci.org/docs/org/

Interested to see your tests ran by KernelCI natively? Here is how to get started with KernelCI.

You already have your own automated execution of tests and would be interested/willing to contribute your results? Please see the KCIDB submitter guide.

You can find the catalog (yaml file) of the tests integrated today here. Some of the most notable test suites reporting today: bpf, kselftest, ltp, perftool, podman, Redhat’s test suites, stress-ng, syzkaller, xfstests, and many more.

If you would like some help, please reach out to the KernelCI community:

Members of the KernelCI Advisory Board (AB) and Technical Steering Committee (TSC) will be present in-person at FOSDEM 2023. Many will be in the Testing and Automation devroom (Sunday morning in room UB4.132).

Looking forward to meeting as many as possible in Brussels, February 4th and 5th 2023.

A case for DAG databases Correlating revision history with CI results

  • Track: Graph Systems and Algorithms devroom
  • Room: K.4.601
  • Day: Saturday
  • Start: 12:00

Growing a lab for automated upstream testing: challenges and lessons learned

  • Track: Testing and Automation devroom
  • Room: UB4.132
  • Day: Sunday
  • Start: 09:30

Rethinking device support for the long-term

  • Track: Kernel devroom
  • Room: UA2.220 (Guillissen)
  • Day: Sunday
  • Start: 16:30

Now KernelCI has a dedicated SysAdmin!

By Blog, Community, News

Recently, we posted about the Request For Proposals to undertake SysAdmin tasks for KernelCI.org. That process is now complete, and we are delighted to announce that the KernelCI advisory board hired Vince Hillier from Revenni Inc to conduct the work.

Vince will work together with the Technical Steering Committee (TSC) to maintain and improve the current project infrastructure. As KernelCI.org scales with more kernels being built and more tests being run, we definitely need help to keep our systems stable and up to date.

The KernelCI project relies on a number of web services which need constant maintenance. These include databases, automation tools and web dashboards for several instances. Some are hosted on dedicated virtual machines (VMs), others in the cloud.

Request for Proposals: Sysadmin Maintenance 2022

By Blog, Community, News

Summary

The KernelCI project relies on a number of web services which need constant maintenance. These include databases, automation tools and web dashboards for several instances. Some are hosted on dedicated virtual machines (VMs), others in the cloud. This Request for Proposals seeks to extend the current team with dedicated sysadmins to ensure the maintenance of these services is being carried out to guarantee a good quality of service. Additionally, some improvements can be made to reduce the maintenance burden.

Budget

We’re expecting quotations for this work package to range between 15,000 and 30,000 USD depending on the contents of the proposal and for a period of six months. Longer sysadmin time available and extra improvements can justify a higher price. Payments would be made via the Linux Foundation using the project’s own budget.

Sending proposals

Please take a look at the full RFP-Sysadmin-Maintenance-2022-v3.pdf document for all the details. Proposals should be sent by email directly to the project members.

The deadline for responding to this RFP is 8th August 2022, six weeks after it has been made public. Then the KernelCI Advisory Board of Members will vote on the 24th August 2022. Exact dates might be subject to change in case of a major practical issue or unavailability of voting members.