Proactive Compaction
for the Linux kernel
Posted on March 3, 2020
(Last modified on August 8, 2020)
| 9 minutes
| 1852 words
| Nitin Gupta
This feature has now been accepted and merged in the upstream kernel and will be part of kernel release 5.9. This post has been updated to match the upstream version of this feature.
In my previous post, I described how on-demand compaction scheme hurts hugepage allocation latencies on Linux. To improve the situation, I have been working on Proactive Compaction for the Linux kernel, which tries to reduce higher-order allocation latencies by compacting memory in the background.
[Read More]
Linux kernel hugepage allocation latencies
A detailed analysis
Posted on February 2, 2020
| 6 minutes
| 1234 words
| Nitin Gupta
Some drivers needs to allocate almost all memory as hugepages to reduce (on-device or CPU) TLB pressure. However, on a running system, higher order allocations can fail if the memory is fragmented. Linux kernel can do on-demand compaction as we request more hugepages but this style of compaction incurs very high latency.
To show the effect of on-demand compaction on hugepage allocation latency, I created a test program “frag” which allocates almost all available system memory followed by freeing $\frac{3}{4}$ of pages from each hugepage-sized aligned chunk.
[Read More]
Google Drive on Linux
Posted on April 4, 2019
| 3 minutes
| 521 words
| Nitin Gupta
There is no official Google drive client for Linux. I tried many different clients found all over GitHub but none of them worked reliably for me except rclone. I also tried third-party proprietary clients like Insync but allowing read-write access to all your Google drive files to a closed source blob is too much to swallow.
Once caveat with rclone is that it does not natively support bi-directional sync (github issue) but someone developed a python script rclonesync-V2 which is a wrapper around rclone which does the job.
[Read More]
Faster compilation with distcc
Posted on June 6, 2010
| 6 minutes
| 1123 words
| Nitin Gupta
Often, you have more than one system at your disposal but no clear way of distributing your compilation workloads over to all or some of them. They might be running different OSes which makes it look even more difficult. In my case, I have one laptop (2 cores) and a desktop (4 cores) connected with a WiFi network. The laptop runs Linux (Fedora 13 64-bit) while the desktop runs Windows 7 (64-bit).
[Read More]
Compressed RAM disk for Windows, The Virtual Way!
Posted on May 5, 2010
| 5 minutes
| 860 words
| Nitin Gupta
Recently, I developed Linux kernel driver which creates generic RAM based compressed block devices (called zram). Being RAM disks, they do not provide persistent storage but there are many use cases where persistence is not required: /tmp, various caches under /var, swap disks etc. These cases can benefit greatly from high speed RAM disks along with savings which compression brings!
However, all this seems to be completely Linux centric. But with virtualization, zram can be used for Windows too!
[Read More]
Setting up KDE API Documentation
Posted on April 4, 2010
| 2 minutes
| 405 words
| Nitin Gupta
Since last few months, I have been playing around with KDevelop which is a KDE based IDE for C/C++ any many other languages. Its a large C++ codebase and navigating through all the files, classes is quite difficult with usual VI + cscope combination. The most lacking part is a readily accessible KDE API documentation which is almost essential, no matter what component you are working on. There is a KDE API reference site but searching there for every reference is very cumbersome.
[Read More]
Comprehensive graphical Git diff viewer
Posted on December 12, 2009
| 2 minutes
| 317 words
| Nitin Gupta
Since a long time, I was looking for a graphical git diff viewer which could show original and modified file side-by-side and highlight the changes. There are few solutions but none of them is sufficient:
A tool included with git called ‘git-difftool’ is partially helpful – it can show changes graphically but diff for each file is shown one-by-one. This is very irritating. In fact, unusable even with just 10-15 files.
[Read More]
Linux kernel workflow with Git
Posted on September 9, 2009
| 6 minutes
| 1130 words
| Nitin Gupta
You worked on some part of Linux kernel. It works great. Now, how to generate the patch series and send it out for review? For this, I always used to generate diffs, create a set of draft mails (one for each patch) in KMail or Thunderbird, and send all these mails one-by-one. This workflow quickly became a big headache. Then I learned Git (and some related tools) to do all this from command line and wow!
[Read More]
Fedora 11 on ThinkPad W500
Posted on June 6, 2009
| 4 minutes
| 844 words
| Nitin Gupta
This was the release I was waiting for! The installation was a bit of a trouble but it was surely worth the effort. Its really the first Fedora release that I really liked. Its stable, has good hardware support and boots really quickly (<20 seconds on my system).
In my case, Windows 7 RC was already installed on one partition, so Fedora 11 (x64) was installed in second partition. So, now I have dual boot configuration – the windows boot loader presents option to continue booting windows or launch GRUB from Linux partition.
[Read More]
ccache to speed-up Linux kernel compile
Posted on March 3, 2009
| 2 minutes
| 355 words
| Nitin Gupta
In case you are unfamiliar with ccache, its a “compiler cache”. Compiling is primarily CPU intensive task. So, ccache caches compiled objects - so next time we compile same code, it reuses these objects thereby significantly speeding-up compilation.
I need to recompile Linux kernel usually several times a day, with different permutations of config settings. This almost forces a ‘make clean’ or ‘make mrproper’ which deletes all compiled objects in build tree and then we have to rebuild everything all over again.
[Read More]