1. Introduction

This guidance will help to:

  • ensure a smooth handover process; from software vendor to software owner
  • standardize software project delivery and deliverables in line with best practice; increasing quality, and normalising communication and expectations across projects to make it easier for teams to work together and transfer existing responsibilities or assume new responsibilities
  • assist product owners who may be unfamiliar with Digital Penang’s approach to delivery and expectations for products.

These standards are designed to help teams follow best practices when designing and building services. They will also be used to ensure that all services meet a minimum level of quality.

Standards work most effectively when they have wide acceptance across the teams who will be using them. Sharing drafts of standards with teams and getting feedback from them frequently is the best way to ensure their relevance. The authors of these standards welcome all feedback; you can find the email address at the bottom of this section.

This standards document is designed to highlight areas of specific importance to Digital Penang. It should not be considered to be exhaustive and should be applied with your best judgement in mind. It can be helpful, for example, to look at other examples of projects and review the pro’s and con’s of different approaches. Most importantly you can always reach out to your product owner and/ or delivery manager for advice on how to apply parts of the standards.

There are complementary standards for digital products and delivery provided by MAMPU (Malaysian Administrative Modernisation and Management Planning Unit) as part of their efforts to modernise the Malaysian the public sector. The Digital Penang standards broadly support the MAMPU standards and only diverge where there is necessity or a refinement to be made. You can find the MAMPU documentation at https://sqa.mampu.gov.my/index.php/ms/. We also link to it in several places throughout this document in order to support Digital Penang’s choices.

Robust standards are particularly important in government because governments provide so many services, it can be difficult to make sure they’re all designed, built and operated in the best possible ways.

You may find that:

  • government is split into different departments, divisions and teams
  • people work in many different locations
  • teams don’t always communicate or know what other teams are doing
  • there’s no shared idea of what a good service is or how to build one

This can mean that:

  • different parts of government build the same thing at the same time
  • services look and feel totally different from one department to the next
  • departments spend money on big digital projects which don’t work or provide a service no one needs

Following standards can help you to:

  • save time and money
  • build services which are simpler, faster and safer for users
  • empower teams across government with shared ideas of good practice.

If you have further questions and/or inputs to the content in this document, feel free to contact Digital Penang at contact@digitalpenang.my

  • <possible information about who to talk to to contribute to these standards ie “If you think something in these standards is wrong or out of date then contact …”>

2. API

We expect software vendors to create a mechanism by which data can be digitally retrieved from the application. This furthers our efforts as an organisation to provide joined up services and make good use of the data our services collect.

2.1 Document your API

APIs need to be documented to be useful. Good documentation will allow the users of the API to quickly integrate with the service and provide value.

We suggest using the guidance here to get an idea of the sections that should be included when formulating your guidance.

2.2 Use REST

Follow the industry standard and where appropriate build APIs that are RESTful, which use HTTP verb requests to manipulate data.

When handling requests, you should use HTTP verbs for their specified purpose.

One of the advantages of REST is that it gives you a framework for communicating error states.

In some cases, it may not be applicable to build a REST API, for example, when you are building an API to stream data.

You can read more about the characteristics of using RESTful APIs for the purpose of service integration in the MAMPU documentation here.

2.3 Use HTTPS

You should use HTTPS when creating APIs.

Adding HTTPS will secure connections to your API, preserve user privacy, ensure data integrity, and authenticate the server providing the API. The Service Manual provides more guidance on HTTPS.

Secure APIs using Transport Layer Security (TLS) 1.2 or above. Do not use Secure Sockets Layer (SSL), TLS 1.0 or TLS 1.1 as these are deprecated.

There are multiple free and low-cost vendors that offer TLS certificates. Make sure potential API users can establish trust in your certificates and make sure you have a robust process for timely certificate renewal and revocation.

2.4 Use JSON

Your first choice for all web APIs should be JSON where possible.

Only use another representation to build something in exceptional cases, like when you:

  • need to connect to a legacy system, for example, one that only uses XML
  • will receive clear advantages from complying with a broadly adopted standard (for example, SAML)

We recommend you should:

  • create responses as a JSON object and not an array (JSON objects can contain JSON arrays) – arrays can limit the ability to include metadata about results and limit the API’s ability to add additional top-level keys in the future
  • document the JSON object/s your API returns. This helps users of your API to be sure they are using the object in the appropriate way
  • avoid JSON object keys that are not consistent for all objects of the same type, such as those derived from data as this adds friction for clients
  • use consistent grammar case for object keys – choose under scores (eg 'email_address': 'example@example.com') or camel case (eg. 'emailAddress': 'example@example.com') and be consistent

2.5 Standardize date / time format

It can be useful to use the RFC3339 standard to represent date and time in your payload response. This helps people read the time correctly.

Use a consistent date format. For dates, this looks like 2017-08-09. For dates and times, use the form 2017-08-09T13:58:07Z or 2017-08-09 13:58:07Z.

2.6 Use Unicode for encoding

The Unicode Transformation Format (UTF-8) standard is used by many governments when encoding text or other textual representations of data.

2.7 Keep a log of requests for personal data

If your API serves personal or sensitive data, you should log when the data is provided and to whom. This will help you ensure ‘privacy by design’ in the delivery of your service. .

2.8 When to authenticate your API

Authentication is required when you want to identify clients for the purposes of:

  • rate limiting/throttling
  • auditing
  • billing
  • authorisation

Your purpose will dictate the security requirements for your authentication solution.

For example, if you need to identify users purely for rate limiting, you may not need to refresh tokens very often as a token in the wrong hands will be unlikely to threaten your service.

2.9 To provide application-level authorisation

Use application-level authorisation if you want to control which applications can access the API, but not which specific end users. This is suitable if you want to implement rate limiting, auditing, or billing functionality. Application-level authorisation is probably not suitable for APIs that will hold personal or sensitive data unless you really trust the end consumers, for example, another department in the same organisation.

We recommend using OAuth 2.0, the open authorisation framework (specifically with the Client Credentials grant type). This service gives each registered application an OAuth2 Bearer Token, which can be used to make API requests on the application’s own behalf.

2.10 To provide user-level authorisation

Use user-level authorisation if you want to control which end users can access your API. This is suitable for dealing with personal or sensitive data.

We recommend using OAuth 2.0, the open authorisation framework (specifically with the Authorisation Code grant type). Use OAuth 2.0 Scopes for more granular access control.

In a more complex environment such as a microservice or federated system OpenID Connect, which builds on top of OAuth 2.0 using JSON Web Tokens (JWT), may also be suitable. OpenID Connect provides a granular and machine readable mechanism to request and receive information about authenticated sessions and end-users in a system.

2.11 Provide ways to monitor your APIs for unusual activity

Your API security is only as good as your day-to-day security processes.

It should be possible to monitor your API for unusual behaviour just like you’d closely monitor any website. Where appropriate you might look for changes in IP addresses or users using APIs at unusual times of the day. The UK has the National Cyber Security Centre (NCSC) guidance on how to implement a monitoring strategy and the specifics of how to monitor the security status of networks and systems.

2.12 When naming and hosting your API

All API naming in URLs (including the name of your API, namespaces and resources) should:

  • use nouns rather than verbs

https://www.api.com/users not https://www.api.com/get-users

  • be short, simple and clearly understandable. Overly long or hard to understand urls may indicate that you should restructure your api or split/ move functionality to elsewhere.

https://www.api.com/users/123/discount not https://www.api.com/get-user-and-calculate-appropriate-discount/123

  • be human-guessable, avoiding technical or specialist terms where possible

https://www.api.com/users/123/discount not https://www.api.com/get-most-recent-pricing-object-for-user/123

  • use hyphens rather than underscores as word separators for multiword names

https://www.api.com/users/123/email-preferences not https://www.api.com/users/123/email_preferences

2.13 When using sub-resources

Sub-resources must appear under the resource they relate to, but should go no more than three deep, for example: https://www.api.com/resource/id/sub-resource/id/sub-sub-resource

If you reach a third level of granularity (sub-sub-resource), you should review your resource construction to see if it is actually a combination of multiple first or second level resources.

2.14 When using query arguments

You should use path parameters to identify a specific resource or resources. For example: https://www.api.com/users/123

You should only allow query strings to be used in GET requests for filtering the values returned from an individual resource, for example: https://www.api.com/users?state=active https://www.api.com/users?page=2.

You should never use query strings in GET requests for identification purposes, for example: https://www.api.com/users?id=123

Query strings should not be used for defining the behaviour of your API, for example: https://www.api.com/users?action=getUser&id=1

2.15 When iterating your API

When iterating your API to add new or improved functionality, you should minimise disruption for your users.

To minimise disruption for users, you should:

  • make backwards compatible changes where possible – your users can ignore properties they don’t expect or understand. This allows you to add fields to deliver new functionality without requiring changes to the users application.
  • make a new endpoint available for significant changes
  • provide notices for deprecated endpoints.

2.16 When making a backwards incompatible change

When you need to make a backwards incompatible change you should consider:

  • incrementing a version number in the URL or the HTTP header (start with /v1/ and increment with whole numbers)
  • supporting both old and new endpoints in parallel for a suitable time period before discontinuing the old one
  • telling users of your API how to validate data, for example, let them know when a field is not going to be present so they can make sure their validation rules will treat that field as optional

Sometimes you’ll need to make a larger change and simplify a complex object structure by folding data from multiple objects together. In this case, make a new object available at a new endpoint, for example:

Combine data about users and accounts from: https://www.api.com/users/123 https://www.api.com/accounts/123

To produce: https://www.api.com/consolidated-account/123

2.17 Set clear deprecation policies

Set clear API deprecation policies so you’re not supporting old client applications forever.

State how long users have to upgrade, and how you’ll notify them of these deadlines.

Your strategy will depend on a number of factors; how many users your API has, whether they are within your organisation or external users, how significant the changes are.

For example if an API is consumed by a single integration from another department in the same organisation it may be possible to approach the integration developers directly to inform them of upcoming changes. However in the case that your API is used by 100’s of users you will need to implement a mechanism for alerting these users. One popular method is to announce deprecation in HTTP responses using a ‘Warning’ header.

2.18 Provide users with a test service

Your API consumers will want to test their application against your API before they go live. If you have a read only API then you do not necessarily need to provide a test service.

Provide them with a test service (sometimes referred to as a sandbox).

If your API has complex or stateful behaviour, consider providing a test service that mimics the live service as much as possible, but bear in mind the cost of doing this.

If your API requires authorisation, for example using OAuth 2.0, you’ll need to include this in your test service or provide multiple levels of a test service.

To help you decide what to provide, do user research – ask your API consumers what a sufficient test service would look like.

2.19 Test your API’s compliance

You should provide your development team with the ability to test your API using sample test data, if applicable. Testing your API should not involve using production systems and production data.

3. Documentation

3.1 Software requirement specification (SRS)

An SRS is used to describe what features are available in the system, this document must be done following user research. This document is able to aid in ensuring the continuity of the project.

This document must be signed off by the stakeholders of the project, so that both the vendor as well as as the project owner will have a clear scope of the project. You can find more information on the beginning phases of a project and agreeing scope in MAMPUs documentation here. You can find more information on the beginning phases of a project and agreeing scope in MAMPUs documentation here. You can also find information on required documents in MAMPUs documentation here.

Here is a template for the SRS with all the necessary sections.

4. Software Development

4.1 Choosing a programming language

The system being built must be written in one of the following languages. Exceptions are allowed where the industry standard deviates from the following list or there are dependencies which require a specific language to be used. (Eg. Python for data analytics.)

4.1.1 Backend

  • Go
  • Java
  • Python
  • Typescript

4.2 Web Frontend Framework

As browsers work with HTML, js and css, this section is only applicable when doing a Single Page Application (SPA). If doing server side rendering, this section can be skipped.

VueJS is the framework of choice. Please discuss with the project owner should there be a need to deviate from it.

4.3 Style guides

Having a consistent style within an organization ensures that different developers coming into the project will have an easier time picking it up, reducing the cognitive load on the developer.

As code style is a matter of preference, we have chosen to lean heavily on the code styles developed by the community of the different languages and frameworks.

You may also want to review MAMPUs guidance on coding style which can be found here.

4.3.1 Go

Code formatting in go is handled by the built in command go fmt.

Aside from that, Effective Go provides best practices on writing idiomatic go code.

4.3.2 Java

Google Java Style Guide

4.3.3 Python

Google Python Style Guide

4.3.4 Typescript

TypeScript Deep Dive TypeScript Style Guide

4.3.5 VueJS

Vue 2 – Official VueJS 2 Style Guide

Vue 3 – Official VueJS 3 Style Guide

4.4 Building with accessibility in mind

Government services must be accessible to everyone. This includes anyone with a visual, hearing, speech, motor or learning impairment. This also includes anyone with a temporary or situational disability, such as a person with a broken arm or working in a loud environment.

Building a service with accessibility in mind not only allows those with access needs to use your service, it also improves the service for everyone else. An accessibility problem with a website can be something that affects everyone, not just people who can only access the web with a keyboard or screen reader.

If your service is not accessible to everyone there can be cost implications too. Users who cannot access your service online will attempt to interact with the service in a way which is more costly for your organisation. (e.g. over the phone or face to face).

4.4.1 How to make your service accessible

Consider accessibility from the start

You cannot achieve accessibility by adding some final touches – it must be considered at all stages of a project. You should review designs for possible issues, write and run tests throughout development, and test services with accessibility in mind.

Understand that not everyone reads content the same way

A sighted user might navigate a page from top to bottom, perhaps skim reading through headings and paragraphs to find the content they want.

Non-sighted users can also skim read a page. Screen readers can announce content by element type, such as headings, paragraphs or links. For example, if a screen reader user expects a page to contain data in a table element, such as a train timetable, they might start by reading through all the tables on a page.

This is why semantic markup and good heading structure are important when building accessible services.

4.5 Documenting decisions

You should record decisions that affect the architecture of your service, in order to preserve the context of your choices.

As agile projects age, it is sometimes hard to keep track of the reasoning behind the decisions made. This is especially true as new people join the projects when those involved in the early stages are no longer around.

It is important to preserve the reasoning so the current team can include it as context when making their own decisions about changes they need to make. For example, understanding whether a particular choice was made for the sake of expediency and can therefore be changed with little impact, or whether there were external reasons behind that decision that need to be factored in.

4.5.1 How to document decisions

Architecture decisions should be stored in version control so there is a record of what was changed, who by, and when. Decisions that affect a specific application should be in that application’s code repository. You may also want to store larger-scale decisions in a central documentation repository.

A suggested format is the Architecture Decision Record, proposed by Michael Nygard in a blog post and since adopted widely. That post describes the format in full, but as a summary it consists of the following sections:

Title

A description of the decision (not the problem)

Status

eg Proposed, Accepted, Superseded

Context

The facts behind the need to make the decision

Decision

What the team has decided to do

Consequences

Both positive and negative consequences of the decision

5. Version Control and Deployments

Source code has to be stored using a version control system (VCS) so that changes can be tracked over time.

It provides the ability to:

  • revert to an earlier version whenever you want to
  • record your changes and the reasons why you made them, to help future developers understand the process
  • work on changes in parallel as a team before merging them together

Git is the recommended VCS. This is supported by MAMPU’s guidance on version control here. In the unlikely event there is a requirement to deviate from this standard please discuss with the project owner.

5.1 Making Commits

Making a commit means making a set of changes permanent. You should:

  • write clear commit messages
  • group changes according to their purpose
  • review new changes

5.1.1 Commit Messages

Writing good commit messages is important. Not just for yourself, but for other developers on your project. This includes:

  • new (or recently absent) developers who want to get up to speed on progress
  • interested external parties who want to follow progress of the project
  • people in the public (remember, we code in the open) who want to see our work, or learn from our practices
  • any future developers (including yourself) who want to see why a change was made

Capturing context around a change allows people to understand why a particular implementation decision was made, much like an architecture decision record. We’re being kinder to our future selves.

Content

A good commit message briefly summarises the “what” for scanning purposes, but also includes the “why”. If the “what” in the message is not enough, the diff is there as a fallback. This is not true for the “why” of a change – this can be much harder or impossible to reconstruct, but is often of great significance.

Example

Set cache headers

prefer:

Set cache headers

IE 6 was doing foo, so we need to do X. See http://example.com/why-is-this-broken for more details.

5.2 Things to keep out of version control

Sensitive information should not be kept in the version control system. These should be provided by the environment which the system is run on, eg. environment variables, secret stores.

Some examples of sensitive information:

  • API Keys
  • Username & Passwords
  • Database connection strings

6. User Research Guidance

Contents

Guidance for doing user research

Setting research objectives

Choosing research methods

All research methods

Guidance for doing user research

Doing research helps teams build services that meet their user’s needs.

User research should take place throughout the development process with insights shaping design and technical decisions.

This guidance helps you to meet the standards issued by MAMPU on:

These standards highlight the need for research activities which include focus groups, observations, interviews and questionnaires to be conducted to ensure successful implementation.

Benefits

By understanding users’ context, concerns and problems what you develop will:

  • be more effective
  • deliver better value for money

User research reduces the risk of wasting time and money building the wrong thing. What you learn from research will inform what you build, without it you won’t know if you are solving the right problem and if it will work well for users.

Including user research methods in service delivery improves quality and reduces costs across your organisation.

The UK government’s “Digital Efficiency report” found that significant savings are made by making it easier to complete tasks online. It estimated online interactions cost the government £0.15 in comparison to £8.62 face-to-face or £2.8 over the phone.

Setting research objectives

Having actionable and specific objectives for what the team wants to learn helps build empathy for their users, and better understand users’ challenges and motivations.

Your team can work together to establish research objectives, a decision maker can decide which objective to prioritise for each round of research.

1. Review the goals

Remind the team about:

  • the outcomes they are trying to achieve
  • the problems you are trying to solve for both the users and the organisation
  • things the team will need to do in the next development phase

2. Identify what you need to learn

Discover what the team needs to learn from the user research:

  • a helpful method is to list down all your questions, assumptions and beliefs held about your users or service you want to test
  • open questions, for example, ‘who’, ‘what’, ‘when’, ‘where’, or ‘why’, are more helpful than narrow or specific ones like ‘do’, ‘have’, ‘will’, or ‘can’
  • involve all team members in this step to get every role’s perspective (senior managers, subject matter experts, developers, UI/UX designers etc)
  • identify things you need to learn, not questions you will ask users
  • a decision maker should decide which objective should be taken forward

3. Choose questions

Summarise the questions, assumptions and beliefs into similar groups, and decide which ones are most important:

  • ask everyone to combine their notes from step 2 together and sort them into groups
  • name each group with an overarching question that encapsulates the theme
  • a decision maker can identify the highest priority questions (usually around 5) that they think need answering first. These prioritised questions are the established objectives for your upcoming round of user research.
  • arrange the groups – and questions within each group – in priority order

4. Choose a research method

Based on the prioritised questions in step 3, decide on the appropriate research method to use and draft a script (also known as a user research discussion guide):

  • Keep a record of all the prioritised questions and de-prioritised questions for future reference.
  • Based on the 5 prioritised questions from Step 3, decide on the methods to use in your research sessions that will help you to answer your identified objectives and questions. See Section 2.2 for suggestions for research methods.
  • It is encouraged to revisit the de-prioritised questions in future rounds of user research so those questions can also be answered at a later stage
  • Draft a user research discussion guide that includes the script, questions and tasks to ask the users who you speak to during the research sessions.
  • Users who participate in user research are known as ‘participants’. They should demographically reflect your product or service’s primary user group.

Choosing research methods

The research process

   
1. Understand the problemWho your users are and what they’re trying to do
  How they do it currently
  The problems or frustrations they experience
  What users need from your service to achieve their goal
  How to define the scope of work
2. Test solutionsImprove the team’s understanding of users and their needs
  Test different design ideas and prototypes with users
  Validate or reject assumptions and hypotheses
  Learn how to build or improve your service
3. Refine and deliver solutionsTest the developing service with users
  Understand and resolve usability issues
4. Ongoing supportAssess people’s experience of using your service
  Understand evolving user needs
  Test new features, changes or improvements to your service

Research methods at each stage

1. Understand the problem

  
Learn about users and their needsObserve people to see how they do things and problems they face
 Use interviews and site visits to explore their lives and work
 Examine existing data and review previous user research
 Outputs:
 i) A journey map that presents the current experience of likely users
 ii) Description of different type of users/personas/typologies
 iii) Defined scope of your service
MethodsContextual Research and Observations
 Creating a User Journey Map
 Personas
 Recruiting Participants
 Analyse Findings
 Report Findings

2. Test solutions

  
To learn more about your users and your design ideasUse interviews and visits to deepen your understanding of relevant aspects of your user’s lives and work
 Try out design concepts with likely users to see how well they meet user needs
 Test interactive prototypes to explore the usability of different designs
 Outputs:
 i) A better understanding of your user’s needs,including their support and access requirements
 ii) Feedback on how well your design works for users
 iii) Helpful insight into usability issues related to layout, functionality and content
MethodsModerated Usability Testing
 A/B testing
 Recruiting Participants
 Analyse Findings
 Report Findings

3. Refine and deliver solutions

  
To learn more about how well your service meets your users’ needsRun private or public beta tests of the end-to-end service with real users, including support options
 Review web analytics and back-office data to measure service performance
 Use surveys or follow-up interviews to collect detailed feedback from service use