Enhancing Shiny Applications for Optimal Performance in the Biotech Industry
Reducing Load Time of a Business-Critical Shiny App from 7 Min to.. 30 Seconds
Here is the story of how the Appsilon team helped an international biotech company
in Shiny apps development and optimization
Appsilon Value Added
Developing 7 Shiny applications that are driving important decisions within an organization
Optimizing Shiny applications to load in just 0:30 seconds, down from 7:00 minutes
Making R Shiny and Posit business critical for the organization
Installing Posit Workbench and Posit Connect
The Challenge
Scientists want to focus on the scientific process and drug discovery. Data Scientists want to support teams with data analyses and insights. Business and Project Management want accurate insight faster for decision-making. Appsilon focuses on developing and scaling applications, letting each team focus on its specialty.
A Promising Partnership and Long-Term Collaboration
The relationship between us, and the client started with an inbound in 2020. Now the Appsilon team extension comprises 6 people with successful ongoing cooperation.
We work with 2 teams; one handling drug discovery, and the other performing translational research (related to genes). In total, we are developing 7 Shiny applications that are driving important decisions within an organization. At the moment, we are developing 4 concurrent applications.
Installing Posit (RStudio) Products
While working with the customer, we identified a need to help with the installation of Posit Workbench, and Posit Connect for one of the teams.
Posit Workbench is a great tool to develop credible insights fast in both R, and Python. The tool is well-suited for enterprise customers who value rapid development, and real-time collaboration. Posit Connect helps data scientists share insights quickly. This strengthens the cooperation with their stakeholders. The successful deployment of Posit products alongside well-designed Shiny applications simplifies teams’ workload.
As a Partner of Posit, Appsilon can show the Proof of Value of Posit products to business departments, helping you secure the budget. Our experience has taught us that developing a successful PoC is helpful to convince the stakeholders and prepare the organization.
Impacting the daily life of our customers
Our client can focus on delivering business value, instead of developing the apps. Thanks to our work, the role and the impact on the business of our client continues to grow . Scientists have faster access to information, data scientists spend less time on app development, and teams have enhanced decision-making tools.
Generating value with open-source R Packages
Apart from developing custom Shiny applications, providing consulting services, and implementing commercial licenses of Posit products, we help customers deliver value with our Rhino, and Data Validator packages.
It’s about a healthy ecosystem
To make sure that our teams are having an impact, we communicate with all involved parties. We collaborate with the Head of Discovery Process, Associate Director, Senior and regular Data Scientists, and Posit representatives to make sure that the work done will help grow the business.
Results
Developing Shiny applications
Productizing the prototypes built by trainees
Posit Connect, and Posit Workbench implementation
R Shiny apps deployed on Posit Connect are business-critical for the customer
Increasing the users of R Shiny
Supporting the better alignment of Business, Data teams, and Posit representatives
“Taking an app that took 7 minutes to load to production quality”
Our client, the company that works on drugs to cure cancer and other serious diseases, created an internal Shiny app. The goal of the app was to visualize clinical data from various studies. The app was created by one person on the client’s side, a Data Scientist. It had thousands of lines of code, it was untested and generally messy; a not uncommon occurrence when an internal project grows in scope.
The application quickly became very important for internal teams of data scientists. Such imperfections like messy code or the lack of unit tests could be ignored, but one thing couldn’t: The application took 7 minutes to load!
Our client asked Appsilon to take over, and fix this. What we found out that the app was that the production version of the app ran on a Linux VM where the data was read from local files. The architecture of this solution is presented below.
Improving this app was a challenge. We needed to ensure the possibility of uninterrupted work on the production version, remove the bugs in the production version when they were reported by the users, improve the app infrastructure, and add requested features. We decided to leave the maintenance of the current version to the client’s team and created a separate branch for all the new changes. As a result, the users would get access to new functionalities only after redeployment in the new, improved version. But this approach helped us avoid the costs and risks related to deployment of the new version to the Linux VM.
More time could be spent on development and getting the Posit (RStudio) Connect version to a point where it could replace the Linux VM version faster.
We moved the app to Posit (RStudio) Connect and integrated it with Profiler (a data lake used by the client). The first version of the new architecture was downloading the files from Profiler to the disk and the app read from there. Then Profiler was updated and with its new functionalities, we were able to update the architecture. The app was fetching the views directly from Profiler, without the need to process the csv files. The study was only loaded when the user selected it (compared to loading all the data at the start of the app in the previous solution). We also introduced the cache.
All the changes significantly improved the loading time for the app.
It depended on the study, but now it was about 2 – 3 minutes. Below is the architecture of this solution.
Iteration 2: Use data lake wisely
We were happy with the results, as the improvement was huge. However, the app still loaded slow for our standards. Our teams were heavily involved in developing the other Shiny applications, so further improvements to this application were not a priority. Until one day, an important stakeholder marked the app as “not production ready”. This meant that we had to find a way to make it faster.
Our teams brainstormed, and came up with four different ideas – one proved to be particularly important.
The final solution used a new functionality of Profiler to access the database directly, instead of using Profiler API. Thanks to this, we could read only the columns that were actually used by the app instead of the whole view as was done before. We also added a progress bar that shows percentage progress to give the user an idea of where the app is in the process. Thanks to the changes, the app took 30 – 60 seconds to load depending on the study.
Below is the current architecture.
