Feedback Collection

This Toolkit for Feedback Collection provides example checklists and questions to help you ensure you have mechanisms in place to ensure you can efficiently and quickly gather feedback (post Go-Live), that allows you to quickly tweak the application and support early operations. In this chapter we will get further into some tips and practices for establishing this process and ensuring effective management. Refer to the supporting material around both the key Roles and Responsibilities and Application Matrix which will be referenced throughout the guidance.

The outcome of this stage should be an updated and prioritized backlog of improvements and features that are ready to be addressed by the next stage (Incremental Improvements).

The primary inputs to this stage are as follows:

• No-code Production Application. The primary input to this stage is the no-code app itself that was just released to Production. While you will have gotten feedback on the app prior to Go-Live, you will start to get broader and even more diverse feedback once the app begins being used by production users. They will begin to exercise use-cases or scenarios that may not have been planned for during design activities. Production app users will also surface feedback on defects or issues that may not have been caught during QA activities; no matter how comprehensive you think your testing approach is, it’s a given that you find latent defects that need to be addressed.

• Existing backlog. Using the guidance outlined in the prior Feedback Collection stage (Stage 6), you have captured a large amount of feedback and triaged what would make it into the MVP release. The items that were deemed not in scope for MVP should still be tracked in your Change Request (CR), log and will be the starting point for you as you add additional feedback.

• Work management processes (e.g. Kanban). Finally, you should continue to leverage the work management process implemented in earlier stages. This continues to be important to allow the no-code team to effectively manage their work items, and also allowing stakeholders to have full visibility of the work being reviewed and where it fits within the ongoing Everyday Delivery. Make sure your No-code stakeholders, and any other power users/SMEs that may be providing feedback, have been enabled with access to your Kanban boards.

• Scope and Change Management processes in place. Finally, as you collect feedback, it is essential to apply rigorous scope and change management. Even post MVP it is essential these processes were established in order to stay disciplined and not lose sight of prioritization.

Before diving into specific approaches for obtaining feedback, let’s begin with some overall and general tips and best practices.

• Stakeholders should already have been identified and engaged during the Feedback Collection stage. This forms the basis for this stage though you should consider if there are any new stakeholders who should be added as the usage of the app rolls out to your user population. It may be important to get more input or feedback from targeted sub-segments of users (e.g. from specific regions or markets), you may not have been able to engage previously.

• Though you may have a number of stakeholders for your app, your No-code stakeholder is essential as they are ultimately chartered with defining success for the app (as defined in the Business Use Case). How do they personally view the app based upon direct use? What feedback have they gotten from their own user or organization? Keep close to the No-code stakeholder (and perhaps cross-functional leaders in other groups that use the app), to keep a pulse on their feedback.

• The intensity of your feedback collection (and response with updates), is important. The pace of gathering and responding to feedback should be maintained from the earlier Go-Live phase. This may include multiple weekly sessions to collect and analyze the inputs and get relevant decisions and approvals.

• As you collect and analyze feedback, ensure it is sufficiently detailed, relevant and actionable. You should go back to the source (either user or system), as needed to make sure the feedback is focused on specific aspects and provides insights that can be used to improve the app.

Much of the work in this Stage is enabled by the No-code team. The table below provides a general framework that can help to form the list of Feedback Collection actions that will help to identify method, source, and responsibility for feedback gathering process.

Task	Description	Responsibility
1. Identify feedback sources	Identify all possible sources of feedback, including both user and system feedback, stakeholder feedback, and internal feedback from team members.	No-code Architect works with stakeholders and team members to identify potential feedback sources.
2. Define feedback metrics	Define metrics for collecting feedback, such as satisfaction ratings, usability scores, or user engagement metrics.	No-code Architect works with stakeholders and the no-code development team to define the feedback metrics that are most relevant for the project.
3. Determine feedback collection method	Decide on the best way to collect feedback, such as surveys, interviews, or focus groups.	No-code Architect determines the best method for collecting feedback based on the feedback sources and metrics identified.
4. Develop feedback collection plan	Develop a plan for how feedback will be collected, including timing, frequency, and who will be responsible for collecting and analyzing the feedback.	No-code Architect collaborates with stakeholders and the no-code development team to create a comprehensive feedback collection plan.
5. Execute feedback collection plan	Implement the feedback collection plan, ensuring that feedback is collected accurately and consistently.	No-code Architect and Creators work with stakeholders and IT / Ops to ensure the feedback collection plan is executed correctly.
6. Analyze feedback data	Analyze the feedback data collected and identify trends, patterns, and opportunities for improvement.	Collaborate with stakeholders and the no-code development team to analyze feedback data and provide insights and recommendations for improvement.
7. Report feedback findings	Communicate feedback findings to stakeholders and the no-code development team, providing actionable recommendations for improvement.	Prepare and present feedback findings and recommendations to stakeholders and the no-code development team.
8. Implement feedback-driven improvements	Work with the no-code development team or fusion team to implement feedback-driven improvements to the project, ensuring that feedback is used to drive iterative improvements.	Collaborate with the development team to implement changes based on feedback and ensure feedback is continually used to inform and improve the project.

If the organization has implemented a Center of Excellence (CoE), the CoE will often play a key role in facilitating the consistent tailoring, use and application of this Feedback Collection framework.

Before your MVP release, you depended heavily on your No-code Stakeholder to identify and prioritize requirements due to their process knowledge and overall business accountability. They remain important but now you need to also gather feedback from the front-line users who use the app frequently (ideally daily). By listening to the end-users, you can collect a much higher volume of feedback and higher quality of insight, as your end-users start vigorously using the app in support of their daily actions.

Here are some tips to consider when working with users:

• Avoid the ‘Squeaky Wheel’ syndrome - the loudest users may not be the best or most representative of the user population. Apply a structured approach to analyzing the overall feedback to prevent the input from just a few users from biasing your view on priorities or decisions.

• Gather feedback from a diverse group - It's important to gather feedback from a diverse group of stakeholders, including users, customers, and team members. This will help ensure you receive a range of perspectives and insights.

• Classify users into critical and causal app users – while collecting usability feedback from first-time casual users is important, they will probably not give you the same depth of insight on functionality as ‘power users’ who have demonstrated frequent activity.

• Not all of your end users will give feedback equally or proactively. Some will love the app and rave about you, while others may hate the app and reject using it. But a lot may fall in the middle, and they may be the hardest to get feedback from – they are likely busy with their daily job tasks, and may not always take the time to offer feedback on areas that were non-intuitive to use, slowed them down, etc. You will have to find ways to proactively solicit this input.

  • Ask specific questions: When soliciting feedback, ask specific questions that focus on the areas you want to improve. This will help ensure the feedback you receive is relevant and actionable.

  • Provide context: When asking for feedback, provide context about the business use case and the specific aspects you want feedback on. This will help ensure that the feedback is relevant and targeted.

• Use a variety of feedback methods: Use a variety of user feedback methods, such as interviews/focus groups, surveys, user testing, and user shadowing. This will help ensure you receive a range of insights and perspectives.

Two of the most common approaches for gathering user feedback include interviews and surveys:

• User Interviews involve having conversations with users or stakeholders to gather detailed feedback about their experiences using the functionality. Interviews can be conducted in any channel. During the interview, it is very important not only to speak about experience, but to go through the system while discussing use case improvements (you may additionally demo the functionality while discussion/interviewing). This approach will add more context and help to gather feedback that is much more precise.

• User Surveys are a common way to gather feedback from a large group of people. Surveys can be distributed via email, social media, or other online platforms, and can be designed to gather quantitative data (e.g., rating scales), or qualitative data (e.g., open-ended questions). Surveys can be useful for getting a broad sense of user satisfaction or preferences.

We’ll briefly outline some of the main steps essential for both of these feedback methods.

User Interviews – key steps

Determine the objective of the interview: Before conducting an interview with focus groups or stakeholders, it is important for the No-code Architect or Creator to determine the objective of the interview. The objective could be to gather feedback on specific features, user experience, or overall project performance. Identify the target audience: The No-code team needs to identify the target audience for the interview. This could include end-users, stakeholders, or both. Knowing the target audience will help the No-code team to prepare relevant questions and ensure the feedback collected is valuable. Develop a list of questions: Next develop a list of questions that are relevant to the project and the target audience. The questions should be open-ended to encourage discussion and provide valuable feedback. Plan the interview logistics: Plan the logistics of the interview, including the date, time, location, and duration of the interview. They should also consider the format of the interview, which could be in-person, via video call, etc, or a combination of both. Practice the interview: The No-code Architect or Creator should practice the interview questions and make sure they are comfortable with the interview format. They should also prepare for unexpected questions or responses. Conduct the interview: During the interview, ensure the focus group or stakeholders are comfortable and understand the purpose of the interview. They should encourage open discussion and listen actively to the feedback provided. Analyze the feedback: After the interview, the No-code team should analyze the feedback and identify any common themes or issues. They should also evaluate the feedback against the project objectives and determine how it can be used to improve the project. (Overall analysis across feedback methods will be explored later in this chapter.) Report the findings: The No-code Architect should report the findings to the project team and stakeholders, highlighting key feedback and recommendations for improvement. While preparing the summary of the interview it is important the do the next: Prepare a summary: Start by preparing a summary of the feedback you have received. This should include the key themes or issues that emerged, as well as any opportunities for improvement or new features that were suggested. Use visuals: Consider using screens, prototypes, to help illustrate the key issues of the UI/UX feedback. This can help stakeholders understand the feedback more easily and make it more impactful. Be transparent: Be transparent about any limitations or challenges with feedback.

User Surveys – Key Steps

Determine the objective: Before creating a survey, determine the objective of the feedback you want to collect. It could be to evaluate user satisfaction, identify pain points, or gauge interest in new features. Knowing the objective will help you craft relevant questions and ensure the feedback you receive is useful. Define your target audience: Identify the target audience for your survey. This could be your end-users, stakeholders, or both. Knowing your target audience will help you craft relevant questions that resonate with them. Choose the type of survey: Choose the type of survey you want to use. Some common types include online surveys, phone surveys, and in-person surveys. Online surveys are the most popular and cost-effective option. Create survey questions: Develop survey questions that are relevant to the project and target audience. Keep the questions clear and concise and avoid leading questions that may bias the responses. Use a mix of closed-ended questions (such as multiple-choice questions), and open-ended questions (such as essay questions). Determine the survey format: Choose the format of your survey. This could be a web form, email survey, or printed survey. Web forms are the most common and allow for easy data collection. Pilot the survey: Pilot the survey with a small group of participants to ensure the questions are clear, and the survey is easy to complete. Make changes as necessary based on feedback from the pilot participants. Distribute the survey: Distribute the survey to your target audience using appropriate channels. This could be through email, social media, or on your website. Provide clear instructions on how to complete the survey and set a deadline for responses. Analyze the feedback: After collecting survey responses, analyze the feedback collected and identify any common themes or issues. Use quantitative data analysis tools like Excel or statistical software to assist in data analysis. Report the findings: Report the survey findings to the project team and stakeholders, highlighting key feedback and recommendations for improvement. Ensure the feedback is used to drive iterative improvements and project enhancement.

User interviews and surveys will still miss important observations, especially if the users are too busy to provide adequate feedback. It is worth also considering techniques like user shadowing - this is a qualitative feedback technique conducted on a small scale, defined sample of users, where an observer watches real-life situations of a No-code app user for a set period (which can be as short as 30 minutes, to a few hours, depending upon the scope of the process). This can help identify insights on improvement areas you might not get from other user feedback channels.

While user-generated feedback is important, it can have biases or end up with gaps – so extracting and collecting system data from monitoring of the Product app is important. This is key as it will help provide a highly objective view on UX and performance, and may help you identify feedback in areas that could go unnoticed at first by end-users. It will also help prevent feedback bias that may be skewed to a subset of highly vocal end-users.

The collection of system-generated feedback may be somewhat specific to your environment and systems, and will most likely take the involvement of resources from IT or Operations. If the organization has implemented a Center of Excellence (CoE), the CoE will also often play a key role in facilitating the design and implementation of automated approaches to system feedback. So you should proactively outreach to the appropriate internal groups early in your project lifecycle (ideally no later than the Project Assignment stage), to begin collaborating with them to implement the right process or technology changes.

While the No-code team may not be implementing these themselves, here are some of the likely types of system-generated feedback you may be able to ultimately take advantage of:

• Analytics data: Analytics data can be used to gather quantitative data about how users are interacting with the functionality. This can include metrics like user tracking, time on site, click-through rates, or conversion rates. Analytics data can be useful for identifying areas where users may be dropping off or encountering issues.

• User Behavior Analysis: analyzing user behavior data to identify trends and patterns that may be impacting app performance.

• Event Tracking: capturing specific user actions within the app and send that data to an analytics platform for analysis.

• Funnel Analysis: tracking user behavior through a series of steps, such as signing up for an account or making a purchase. By analyzing the data collected at each step, you can identify where users are dropping off and make improvements to the user experience.

• User Segmentation: dividing app users into groups based on specific characteristics, such as demographics, behavior, or usage patterns.

• Error Tracking: Error tracking involves logging and analyzing errors that occur within the app, such as crashes, timeouts, or other system errors.

• Latency Analysis: Performance or Latency analysis involves measuring the time it takes for the app to respond to user actions or requests.

This data will need to be collected from a variety of sources – website monitoring tools, performance monitoring tools, system/logging databases, application databases, and from the no-code platform itself. It can take a while to pull this information together, so consider this an ongoing goal – it need not all be in place on day 1. However, over time this type of system feedback can be immensely valuable in driving insight you may not get from any other sources.

Between the combination of user and system-generated feedback, you will be collecting a vast amount of information. Don’t lose the forest for the trees - analyzing feedback is an essential part of the feedback process. It involves reviewing the feedback you have received and identifying patterns, trends, and insights that can be used to improve the project.

The sample matrix below provides a reference framework for categorizing feedback based on the type of issue, severity, and priority. The categories are divided into user experience, functionality, support, user engagement, and business impact. The feedback types include usability, design, accessibility, performance, features, bugs, documentation, customer service, technical support, content, interactivity, business value. The severity column indicates how severe the issue is, with high severity issues being the most critical. The priority column indicates the priority of each issue, with high priority issues being the most important to address.

Category	Feedback Type	Example Issue	Severity	Priority
User Experience	Usability	Navigation is confusing	High	High
User Experience	Design	Font size is too small	Medium	Medium
User Experience	Accessibility	Color contrast is low	High	High
Functionality	Performance	App is slow to load	High	High
Functionality	Features	Missing feature X	High	High
Functionality	Bugs	Button not working	High	High
Support	Documentation	User manual is outdated	Medium	Medium
Support	Customer Service	Long wait times on support calls	High	High
Support	Technical Support	Bug was not resolved	High	High
User Engagement	Content	Content is outdated	Medium	Medium
User Engagement	Interactivity	No way to interact with other users	High	High
Business Impact	Business value	Functionality is not generating enough revenue	High	High

Some final tips:

• This example framework will help you understand the high-level overall feedback (across different feedback types and formats), and identify any common themes or issues. Use quantitative data analysis tools like Excel or statistical software to assist in data analysis.

• You may also spot gaps or over-concentrations of certain types of feedback, which may help you go back to your users or system data to expand gathering of feedback in areas that may have been initially underrepresented.

• You’ll need to also make sure you have the supporting detail behind each of these feedback items; you may need to go back to the feedback source(s) to gather more input/context, and ensure it was sufficiently detailed, relevant, and actionable.

• Finally, you should report out the overall findings to the project team and stakeholders, highlighting key feedback, overall insights, and recommendations for improvement. This will be used by the next Stage 10 (Incremental Improvements to drive iterative improvements and enhancements to the application.)