Designing a Big-data applications management tool for data engineers
Ocean for Apache Spark
Overview
Spot by NetApp is a CloudOps company helping data engineers automate and optimize their big-data applications. My role was to improve the experience and process of managing configurations of big data applications.
As the Lead UX Designer for the Big-data product, I managed this project's user research and user experience design from start to finish with full collaboration between the design, engineering, and product teams.
What the #*% is it?
What are Config-templates?
- Data engineers use Apache Spark to run big-data applications. Each type of application needs its own configuration (in a JSON format).
- The Config-templates page is a workspace within the Spot console for storing and managing them.
How do users use them?
- Select a template and copy its code.
- Paste it in Apache Spark environment and run the app.
- Come back to Spot's console to view the app's progress, stats and issues.
Motivation for the redesign
Improving usability issues of the current design:
- Template ID is visible only for the selected.
- Viewing few amount of templates - only 3 on small screens! makes it hard to browse between templates.
- Adding more info to each template such as last editor, number of versions and tags will make each card much more overloaded.

Original config-templates screen
What do we want to achieve?
Research
Initial Discovery
We began with a solid foundation:
- Existing user personas from active customers.
- Usage data from current config-template features.
- Internal subject matter experts.
Who are the users?
Data engineer
Designs, builds, and optimizes data pipelines on Apache Spark, ensuring efficient data ingestion, transformation, and processing for analytics and machine learning applications.
"I need things to be very organized so I can find what I need easily and focus on my tasks"
Data Platform Engineer (DevOps)
Develops and maintains the infrastructure for running Apache Spark applications, managing scalability, performance, and reliability to support seamless data operations.
"I need an option to view what changes were made in the templates over time and the reasons they were made"
Methods
Surveys
- Open-ended and closed questions.
- Features rating and ideas ranking.
- Gained qualitative and quantitative insight.

Survey results
Customer & Stakeholder interviews
- Explored their daily workflow for creating and maintaining config-templates.
- Examined integration with existing tools and processes.
- Assessed work aspects both within and outside our product.
- Analyzed current usage patterns and desired capabilities.
- Studied user interactions and workflows.
- Uncovered key pain points and frictions.
- Identified short-term and long-term objectives.

User Journey Mapping
- Collaborative workshops with cross-functional teams.
- Identified key interaction points and friction areas.
- Prioritized problem spaces.
- Created alignment on solution requirements.

Key Research Findings
"As a data engineer I want to:"
- Filter and sort templates based on various criteria so that I can focus on relevant items.
- Organize and filter config-templates more effectively.
- Group templates by different attributes so that I can better organize my view.
- Perform bulk actions (like editing tags or deleting) on multiple templates at once so that I can manage them more efficiently.
- Compare templates side by side so that I can easily identify differences for debugging.
Insights
- Users strongly desired version control and change history.
- Template organization was a major pain point: Need for logical grouping and categorization and no way to track template purposes, and uses per application.
- Search and filtering capabilities were crucial for large template libraries.
Ideation
Design solution
So, we had to do some prioritization based on development capabilities
- Add version history and changelog option.
- Tagging system - This feature did not exist in the Spot console, and deciding on its behavior was challenging.
- Advanced search and filtering.
- Template comparison tools.
- Applications running on config-templates list.
- Presets for templates with common configurations.
- Bulk actions for template management.
The one page improvement task evolved into an enhanced design and development project...
Design process
How should we approach each feature's design
- Analyzed existing design system components.
- Researched applicable design patterns.
- Should we add a draft mechanism for unfinished editing.
- Should commenting on a changelog be mandatory.
- Mechanism of the tagging system.
- Low fidelity wireframes on the page layout.
- Original Cards layout vs a new table design.
- Working with tags.
- History view - compare and changelog.
- Presets - custom and built in use cases.
Editing

Tagging Process

Exploring layout directions
- The new features were added to the cards design. Several attempts and iterations of various layouts were made, such as wider cards to hold more data, an expanding panel and a table.
- Several user' interviews and testing were conducted with internal users and with customers.
- The testers were presented with the various layouts, and were asked to perform tasks such as filtering, sorting and performing bulk actions.
Expanded panel
By default: The template’s content is collapsed.
Clicking the expand icon expands the panel over the config-templates list.
- More space for browsing templates
Pros:
- Can view the template content at first look
- Not a usual design pattern
Cons:

Cards design - Actions popup
Issue:
The placement of bulk action buttons outside this area could be confusing.
Solution:
Actions popup - Triggered by checking an item
- Selecting a template opens a modal with options.
- Selecting 2 items enables comparison.
- Saves space on the page
Pros:
- No indication for the actions without checking an item
Cons:

User testing insights
Card design
- Intuitive grouping by attribute - Tags, Author, Cluster
- Template view is wide from the start
Strength:
- Sorting is limited
- Only a few templates are on screen
- Added more info to each template makes the cards much more overloaded and harder to browse
Weaknesses:

Table design
- Sortable by multiple parameters (ID, name, cluster, author, date)
- No need to hide config-template info
- Easier template browsing
Strength:
- Less initial space for templates (solved with full-screen option)
- Grouping is limited to column headers, missing tag-based grouping (design system)
Weaknesses:

Instead of the design system’s grouping option I switched to a simple “Group by” selection

Final design & New UI
Main page
I have created high-fidelity mockups and ready-for-dev Figma files, considering all edge cases, empty states and placeholders. I also had the opportunity to use Spot's newly developed design system.


Behavior of the template panel for multi select.
The compare button is enabled when two items are selected.

New template popup
The user can select to start from a preset, duplicate a template or start from scratch.

Presets page
In order to aid selecting the right preset, the user can:
- Select from some common starter use-cases
- Select from a user created preset specific to their needs

Tagging system

So, how did we do?
Monitored via Pendo
Monitored via BI tools
Monitored via Solution Architects
Monitored via sales reports
Key takeaways
The config-templates redesign project presented both challenges and learning opportunities, which are the reasons I find UX work so fulfilling. Through this experience, I developed a deeper understanding of the pain points faced by this product's users. The project also enhanced my ability to effectively communicate with colleagues and stakeholders - a valuable skill for all projects.