#8 No Code: The Revolution that wasn't
on Studio Ghibli and Nvidia Ampere architecture whitepaper
Every few years a startup comes along promising to allow building software products magically without writing any code. The idea has a basis. Developing new products is slow, painful, and expensive. Not to mention the numerous bugs, multi-platform support, scalability, security issues, and regulatory compliances to take care of. If only there was a way for non-programmers to quickly transform current manual business processes into apps that is compliant and secure and deployed without any extra effort (and maybe works on all mobile and web platforms.)
AWS released its no-code platform Honeycode last week. The hype around it was expected. Before moving further, let me introduce you to something called a Citizen developer. To be honest, I did not know about this term until 2019. Who else but Gartner proposes a definition (and then laughs their way to the bank) -
A citizen developer is a user who creates new business applications for consumption by others using development and runtime environments sanctioned by corporate IT.
No code solutions typically operate on three levels of abstractions -
Common business processes - Departments in large enterprises might have similar requirements. For example, they might need an app where employees can request equipment based on some set of rules, or a quick unplanned app for an upcoming conference, and so on.
Common components - Think of all the user interface components that you interact with daily. There are form elements like dropdown and text boxes to accept information, maps that visually show somethings like in Uber, charts, UI elements to show progress, spreadsheets, etc.
Common low-level operations - The gist of a majority of enterprise applications is a way to store or request for data, then apply a bunch of data manipulation functions on either single data points or aggregated metrics, and then a way to show the data. You could add authorization so that a specific set of people can write or access certain data. If only there was a way to provide a datastore and these primitive operations in a drag and drop interface where you could design your rules and apps visually.
There have been several attempts at all three levels in the past, most of them failing.
Partial win: The rise of Low Code
While No code has not been able to make significant breakthroughs, Low code is making inroads in larger enterprises. The subtle difference is that instead of replacing programmers in entirety, low code augments programmers, automates or delegates repetitive CRUD (Create, Read, Update, Delete) app development to business analysts or subject matter experts, while sometimes even providing the ability for programmers to use their existing tools and processes along with the Low code platform.
Enterprise Low code platforms like Powerapps by Microsoft and Lightning platform by Salesforce are some such examples. You could go from a spreadsheet to a functional app taking in data and displaying charts and output in a matter of days. This also gives the Low code platforms a fair bit of understanding about the amount of money they are saving for their customers, and hence the ability to price accordingly. Microsoft Powerapps recently created and launched a crisis communications app template for COVID that can be used by companies using the platform to roll out their whitelabel app within days.
Some of the other common tasks that previously required a developer to write some code include automated extraction of data from documents based on certain conditions (e.g. Government PDFs, financial reports, Invoices, Resumes, etc.), object detection in traditional industries like retail, integrations between different pieces of software that the organization is using. There has been a rise of an entire army of Robotic Process Automation (RPA) startups based on the advances in machine learning in the last decade to focus on these problems.
Then there are non-enterprise focused, SaaS Low code platforms. Zapier allows you to add integrations amongst multiple SaaS products. Airtable is a hybrid between excel and a database. You can create professional-looking websites without code using Webflow (one of the things startups are using for quick landing pages). Bubble.io allows you to create web applications without writing any code. Coda is smarter Google docs, sheets, Trello, and more. Glide claims to create an app from a simple spreadsheet in five minutes. There are deep learning-based tools that could take a wireframe and convert into an app skeleton or a simple webpage.
One of the Low code trends is to abstract complex stuff and hide them under an API. AutoML deployments by companies such as Google solves a lot of common machine learning use cases which can be simply used by say a web developer. Microsoft AI builder providers a visual interface to integrate intelligence into your apps and create custom machine learning models with your data.
Low code solves some of the typical problems of the software development life cycle. Reusable components are already tested, so unit testing at granular levels can be done away with. The integrations can be checked for potential violations of compliance policies in an instant versus manual checks in case of software developed from the ground up. Similarly, best practices for security are already taken care of. Platforms such as Salesforce Lightning or AWS Honeycode also provide instant scalable deployment thus doing away with a lot of operations headache.
Low Code is increasingly being applied to platforms such as IoT where a lot of functionalities and use cases can be mapped to a few reusable components. Some companies are even going ahead with providing the standardized hardware to collect the data and then provide their platform to use the collected data and build apps on top of them. It is interesting to note that while at the same time there has been a huge push towards teaching everyone to code, there is a similar movement going on to ‘democratize’ building apps.
The wait for the next abstraction
There is promise in abstracting duplicated code into components that can be reused without additional effort. However, I am not sure about the direction that we are going in right now. Integrations with components still require a programmer and it might just be as good to an equivalent editor or IDE which auto-fills components instead of small chunks of code.
We have moved from assembly to high-level programming languages primarily to increase productivity for a long time. The next such jump might never come. I like to think in the way of communication languages. Are the languages we use for communication the most efficient way to convey our thoughts? Possibly not. But since it is so widespread now, there are lesser chances of adoption of an abstraction built on top of say English.
Studio Ghibli and the art of hand drawings
The phenomenon of No code is not only relevant to programming but also art. With significant improvements in computer hardware especially GPUs, animation studios have been moving towards computer generation of graphics. One of the most stubborn studios in this regard is Studio Ghibli. Ghibli still primarily relies on hand-drawn graphics, with one of the most famous patrons, Hayao Miyazaki insisting on checking all hand-drawn frames himself. There is a limited amount of computer graphics involved.
I have a strange relationship with anime. The typical conversation on this topic with someone goes like this -
Someone: Do you watch anime?
Me: No.
Someone: Have you watched X?
Me: Yes.
Someone: watched Y? Z?
Me: Of course. And A, B, and C too!
Someone: Miyazaki 😍
Me: Ghibli ❤️
It is strange that sometimes you do not talk about a topic with anyone for a few years and then in a week, you discuss the same topic with multiple people. As an outsider, if I close my eyes, mention of Japan brings the following in mind - Sushi, Bullet Trains, 1945 bombing, Murakami and Studio Ghibli.
The first Studio Ghibli movie for me was Miyazaki’s Spirited away. I was a 17 years old sophomore still protected from the real world by my parents. And here I was watching a brave 10-year-old trying to get her parents back. Then over the last decade, I watched them all. Totoro, Ponyo, Kiki became some of the favorite characters. I would revisit the movies years later and the movies were as beautiful and fresh as if I was watching them for the first time. Except for Grave of the fireflies. The only Ghibli movie that I have perhaps not watched again. The same reason why no one watches Requiem of a dream for a second time. While Miyazaki gets a lot of attention, it is perhaps worthwhile to mention that two of the Studio’s best movies - Grave of the fireflies and Only yesterday- were driven by Isao Takahata who passed away in 2018. I somehow connect more with Ghibli than say Pixar movies. One of the negatives is that Ghibli and especially Miyazaki focuses on the beauty of people and the world most of the time, ignoring the terrible and wretched characters that we get to see in our life from time to time.
Here are my favorite Ghibli works, as far as I could recall -
Spirited Away
Grave of the Fireflies
Ponyo
Kiki’s Delivery Service
Porco Rosso
Other must-watch: My Neighbour Totoro, The Wind rises, Only yesterday, Princess Mononoke
Miyazaki’s movies feature the innocence of the characters, the overall feeling of warmth and lack of gloom in all the scenes. There are no weak frames. The entire movie is a visual treat. Even things like the sea and the wind feel alive. Perhaps there is a subtle message hidden somewhere in all the movies. In Ponyo, a lead character named Sosuke says we should not judge people based on their looks, Spirited away and the theme of greed, the innocence in Totoro where two little girls befriend a friendly giant spirit amongst others. A movie that does not gets talked often about is Only yesterday. Then there is the music. The background score and the lyrics to the songs, which retain their charm even after translation, for example, this one from Spirited Away -
Somewhere, a voice calls, in the depths of my heart
May I always be dreaming, the dreams that move my heart
So many tears of sadness, uncountable through and through
I know on the other side of them I'll find you
Everytime we fall down to the ground we look up to the blue sky above
We wake to it's blueness, as for the first time
Though the road is long and lonely and the end far away, out of sight
I can with these two arms embrace the light
As I bid farewell my heart stops, in tenderness I feel
My silent empty body begins to listen to what is real
The wonder of living, the wonder of dying
The wind, town, and flowers, we all dance one unity
Somewhere a voice calls in the depths of my heart
keep dreaming your dreams, don't ever let them part
Why speak of all your sadness or of life's painfull woes
Instead let the same lips sing a gentle song for you
The whispering voice, we never want to forget,
in each passing memory always there to guide you
When a miror has been broken, shattered pieces scattered on the ground
Glimpses of new life, reflected all around
Window of beginning, stillness, new light of the dawn
Let my silent, empty body be filled and reborn
No need to search outside, nor sail across the sea
Cause here shining inside me, it's right here inside me
I've found a brightness, it's always with me.
Recently, I was able to watch a four-part documentary on Miyazaki, perhaps the only one that looks into his life over the years. (It is free to stream here.) Some of the points that stood out - Miyazaki starts by drawing scenes that he wants to be in the movie somewhere, rather than starting with a screenplay or a story. In an era where everyone is selling some productivity tips on how to be busy the entire day, it is refreshing to see him ‘wasting’ weeks doing nothing and them coming up with something beautiful. The animation frames of Ponyo were hand-drawn despite the technology being available. It is probably still the case with other movies. Even someone like Miyazaki is concerned about his image when he says - ‘I want to make a film that won’t shame me.’ Imposter’s syndrome can strike the best even after several decades of work. Miyazaki’s mother was chronically ill for a long time till her death and he remained craving for simple joys like a hug, one of the reasons he tries to incorporate a character modeled on her mother in his movies. He says - ‘Ponyo is for kids who wish they had not been born.’ Miyazaki shares a complex relationship with his son Goro, who is also a filmmaker, though not as accomplished yet. The series is recommended for any of Studio Ghibli/Miyazaki fans. I liked a Ponyo song that Miyazaki hums -
Because my ride has yet to come,
In the mean time, please let me take a walk
To dance again just one more time,
Become a gentle breeze.
Miyazaki is currently working on what would be his last movie. As he says - ‘I want to end on something good.’ A lot has changed in tech since Nausicaa came out 36 years ago. Computer rendering using modern GPUs have reached a point where a desktop can be used for realtime raytracing with stunning results. Ghibli is still driven by animators drawing out most of the work with minimal CGI and possibly will be for a long time.
Reading the Nvidia Ampere whitepaper
I had previously written about the Nvidia Ampere GPU architecture launch. Yesterday, I could read the Nvidia Ampere architecture for their upcoming GPUs for datacenters and possibly other product lines. There are some significant improvements over Volta in hardware, additional instructions, and compute. You can download the annotated paper from my Dropbox here. Here are some notes from the paper that I found useful in comparison to Volta, the last major datacenter refresh - (You might want to skip this section if GPUs or computer architecture do not interest you. Alternatively, let me know if you would like me to write about GPU architectures from the basics.)
Physical characteristics
7nm vs 12nm Volta (better power efficiency more dense transistors, more than double at 54.2 billion)
1.5x more L1/shared memory per SM vs 96KB in Volta, 2x more L1 cache bandwidth
40 MB L2 vs 6MB in Volta. Benefits training with small batch sizes.
SXM4 form factor v SXM2
Almost double the DRAM bandwidth (HBM2)
Simultaneous execution on FP32 and INT32 cores in each SM, same as Volta
ECC memory for DRAM, caches, and register files, as usual for datacenter devices.
Multi-Instance GPU (MIG) support for Cloud Service Providers
A GPU can be divided into up to 7x more GPU smaller instances as if they were separate physical GPUs.
Isolated paths for the entire memory system- isolated crossbar ports, L2, memory controllers, and DRAM address buses.
Migrate vGPUs between GPU instances on a single GPU or to a different GPU in a cluster. The state information can be saved and restored.
Sparsity support
2:4 sparse matrix, two non-zero values in every four-entry vector.
Nvidia recipe - Train normally, fine-grained structured pruning, fine-tune remaining non-zero weights.
Sparse MMA operation (Matrix Multiply-Accumulate) 2x speedup with N/2 compute cycles versus normal matrix multiply.
New Floating Point formats
TensorFloat-32, BFloat16/FP32 mixed-precision Tensor Core operations. TF32 allows normal FP32 DL training acceleration in TF32 without any code-changes. Up to 10x throughput compared to FP32 in V100.
TF32 8-bit exponent (same as FP32), 10-bit mantissa (same as FP16) and sign-bit.
Compute Data Compression
Compress data to save bandwidth between DRAM and L2 (up to 4x R/W bandwidth)
Compressed data save 2x L2 capacity
Compression between L2 and shared memory saves up to 4x L2 read bandwidth.
Scaling up the infrastructure for multi-GPU training
Faster NVLink 3 (2x than NVLink 2)
Mellanox Infiband and Ethernet (200Gbps)
PCIe Gen 4
Better architecture features - Page faults at remote GPU are send back to source GPU using NVLink.
CUDA improvements
Async Copy from DRAM bypassing cache copy dance and Register File while simultaneously doing computation with SMs.
Async barrier (sync threads at different granularities rather than just warp or block). An async barrier can allow using of time when a thread arrives at a barrier much earlier than others rather than just waiting.
Task graphs to improved launch and running overhead.
L2 cache residency control. You can persist data in L2 programmatically to prevent eviction.
New warp reduce instruction with hardware-accelerated arithmetic.
Tensor Cores Improvements
2x compute for FMA (Fused Multiply Add) per SM than Volta and Turing
Acceleration for all data types FP16, BF16, TF32, FP64, INT8, INT4, Binary.
Additional specialized hardware
A new 5-core specialized JPEG decode engine, since CPU and even GPU cores are not better at this task.
Optical flow and stereo disparity accelerator for vision, automotive, and navigation tasks.
Five Video Decoders compared to a single one in Volta.
Other things that matter
From Apple’s WWDC 2020 - iOS 14 is coming with some significant interface changes. Funnily, a new feature that reports to users when an app copies their clipboard data exposed TikTok and other apps as doing so. iPadOS now supports deeper search and handwriting recognition for Apple Pencil anywhere on the device. macOS Big Sur arrives with the version bump to 11, no more OS X. Apple announced its silicon chips based Macs to start coming out late this year. FaceID and TouchID based authentication can now be used by arbitrary websites. The fastest supercomputer on the planet is now based on Arm and located in Japan. Apple has acquired Fleetsmith, an Apple devices management platform for corporates. YCombinator is cutting down its standard investment deal from 150k to 125k for 7% equity. YC plans to support more companies worldwide, as is evident from their Indian representation with more than 30 Indian companies last year, compared to 1-2 per year five years ago.