Sections
You are here: Home Response Monitoring and Evaluation

Monitoring and Evaluation

Table of Contents
  1. Introduction
  2. Definition
  3. Objectives
  4. Customer
  5. Delivery Performance
  6. Conclusion
Annexes
Not synchronized for offline browsing This Chapter is available for offline browsing

Introduction

The Logistics function in humanitarian organisations is made up of people, processes and systems working together to support efficient and effective delivery of services. Controls are normally put in place to monitor weaknesses, poor designs in projects and improper implementation of programs. Based on continuous monitoring, these weaknesses or shortfalls against targets or objectives set can be corrected or revised in order to continually improve performance, thus reducing the risk of exposure and strengthening the response to needs.

Monitoring and evaluation are integral parts of management and provide a link between planning and implementation.  While monitoring focuses on the activities and outputs, evaluation focuses on the outcome and goals.

Monitoring is initiated at the beginning of a programme, project or emergency response and built into the design, assessment and planning phases of the logistics aspect. It focuses on inputs and outputs and basically tracks and assesses implementation of the logistics aspect of the programme, project or emergency response. It is the continuous process of gathering logistics and programme information to measure against pre-set key performance indicators (KPI's), benchmarks or previously base-lined indicators that are aligned to the goals and objectives of the program.

Evaluation, like monitoring, is a continuous process. The evaluation of the quality of the output should be undertaken in such a way that shortcomings can be identified and corrected. Evaluation should also feed into the planning process continuously so that the planned method of the intervention can be modified to take into account the realities and conditions on the ground. Evaluation provides a tool for management to ensure that focus is maintained.

Definition

For logisticians, monitoring and evaluation may be defined as follows:

Monitoring: to review on a continuous basis the degree to which the logistics activity is completed and if targets are being met. This allows corrective actions to be taken.

Evaluation: to analyse progress towards meeting established objectives and goals. It is done on an ad hoc, monthly, quarterly or yearly basis. Evaluation provides feedback on whether plans have been met and the reasons for success or failure. It should also provide direction for future plans.

Together monitoring and evaluation provide the basis for performance management. Also key to performance management is aligning performance metrics to the goals and objectives of the program.

Without alignment, monitoring and evaluation is ineffective as the direction in which the program is heading is unknown and therefore M&E is unfocused and potentially disruptive to the program.

Without alignment you do not know what to monitor or how to evaluate it.

Objectives

Monitoring and evaluation has several purposes:

  • to provide information to users on the service level they can expect;
  • make an objective evaluation of services and activities;
  • identify problems in the supply chain;
  • determine what measures are needed for improving services;
  • understand the need to increase or decrease resources;
  • objective measurement for calculating reorder levels;
  • define parameters for the periodic review system calculations;
  • evaluate performance of individual staff members;
  • motivate logisticians.

Quantitative and qualitative measures can be monitored for the entire supply chain, from the manufacturer to the beneficiary, as well as for individual parts of the supply chain. Every link in the supply chain should be seen as a service receiver (from suppliers) as well as a service provider (to users). The management of stocks and the quality of services received will affect the quality of services provided. It is important to make this distinction when there is a requirement to improve service provided to the final user or consumer.

Customer

For the purpose of this document, a customer is the final user or the identified final beneficiary of the logistics service. There are two sets of customers: internal and external customers. The internal customer is the user or beneficiary department within the same organisation and the external customer are the beneficiaries in the communities that we serve. 

The Monitoring and Evaluation of Services  

There are two main points to consider when monitoring and evaluating order management:

  • the services provided to customers (quality of service to  customers, customer perception of service);
  • the services received from suppliers (including upstream logistics centres).

Services provided to customers

This point can be looked at from the perspective of the order management team providing services to customers/users/beneficiaries, as well as the customer's perspective of the effectiveness of services received from the Order Management team. The quality of service provided to customers needs to be measured and monitored as a basis for management and planning. The most used indicator in this regard is the percentage of orders delivered in full, on time and error free (DIFOT).

On-time Orders received on or before the date requested
In-full
Orders are complete in quantity
Error-free Orders are complete with proper documentation, labelling, and without damage to items or packaging

The imperative for getting it right

While suitable performance measures provide the foundation for informed decision-making, improper performance measures can distort the conclusions drawn and badly impact efficiency by effectively disguising critical issues and warning signs. 'Good' performance measures have several distinguishing characteristics:

  • they are directly related to objectives and strategies;
  • they must be understandable but not under-determining;
  • they must be meaningful;
  • they vary between locations and customer segments; and
  • they provide fast feedback.

Additional key indicators:

  • information available for customers (items, lead times, order status, etc.);
  • response time (order acknowledgement, queries, etc.);
  • number of claims and items returned;
  • number of stock-outs;
  • number of back-ordered lines; and
  • average backorder time.

Customer perception of service

Logisticians have priorities in their day-to-day work, however it is important to keep in mind the objective: to provide excellent service to the customer, with consideration to their objectives.

An important aspect of measuring the customer's view of the services provided is through a customer satisfaction survey. This is done in the form of a questionnaire. Indicators that are quantitative and objective monitor "hard facts" such as inventory availability and lead times. The purpose of requesting feedback from customers is to gather subjective information that cannot be measured. Customers are asked to rate the importance and quality of services received to determine their perception of logistics services (qualitative analysis).

The Customer Satisfaction Survey Questionnaire

A useful tool that can be used by the logistician is the Customer Satisfaction Survey Questionnaire. It can be used to:

  • assess satisfaction of customers with logistics services;
  • determine services which are of the greatest importance to the customer;
  • identify services which are not considered necessary by customers and may potentially waste logistics resources which can be more useful elsewhere;
  • detect changes in the service needs of customers, programs, or overall situation changes;
  • as a monitoring tool after implementing measures to improve customer service to evaluate their success;
  • debrief  all  staff,  and  headquarters,  on  the  performance  of  the  logistics department, and
  • to be effective, this survey should be carried out on a regular basis.

Services received from suppliers (including upstream logistics centres)

The same measures can be used as for the "service provided". The main difference being that the Logistician becomes a customer of logistics services.
The number of "stock on order" items and "stock-outs" are crucial indicators as they relate to issues that have an impact on the services provided by order management.
Additional key indicators are:

  • order status information (initial acknowledgement, regularity, frequency);
  • technical support (product information, quotations, repairs);
  • stock availability;
  • document quality;
  • quality of items;
  • quality of packaging (damages during transport);
  • treatment of claims (response time).

Delivery Performance

Monitoring Performance

Performance measures provide a management framework, facilitate communication, direct behaviours within the organisation, foster improvement and assess positioning and operational capacity.

There are a great number of indicators for measuring the quality of services along the supply chain. These include time, scope, service and cost. No single indicator allows measuring all aspects of supply chain management. They should be interpreted and used in combination.

Performance of the supply chain should be measured regularly, consistently and systematically rather than waiting until services have deteriorated below acceptable levels. Monitoring should be a key element of running a logistics service.

Actual performance should be compared with the goals and objectives established for the program, which defines the level of service logistics tries to achieve.

Some aspects to monitor in logistics

Supply chain response/lead time

Lead time is the time between placing an order and receiving the goods or service. Delivery too early or too late may also incur unnecessary costs. Delivery too early can mean goods have to be stored until needed and will incur costs whilst being stored in warehouses. Delivery too late can mean the costs of setting up facilities, for example feeding stations and having people ready to distribute goods, is wasted because the goods have not been delivered. It can cause the organisation to incur additional transport costs, for example, aircraft/helicopters have to be used to move the goods more quickly along the next part of the supply chain. In disaster/emergency relief situations, timing of delivery can have a serious impact on the relief operation and on the beneficiaries.

Information on status of orders

The internal performance of the logistics function is dependent on the efficiency and effectiveness of each of the logistics components. For example, one performance indicator for procurement, would be its ability to disseminate information on the number of orders issued. This will enable the warehouse to provision for storage space. Unexpected deliveries can disrupt operations and put the stock at risk of being stolen when left in the open.

Efficiency

The measurement of efficiency is sometimes relative and dependant on what an entity defines as efficiency. In this context, efficiency is the satisfactory delivery of a logistics service that enables the end user to fulfil the intended purpose of the request. A good example is the request for malaria prevention medication ordered to be pre-positioned before a malaria season. A late delivery would mean higher incidents of malaria and an increase in the request for malaria treatment rather than malaria prevention drug.

Total supply chain costs

The total cost concept focuses on reducing the total cost of logistics rather than the cost of each activity. An organisation should monitor the cost reduction across the board and evaluate the impact on each of the logistics components. For example, purchasing in bulk may reduce the cost of the product but at the same time increase the stock holding costs.

Inventory costs

Inventory carrying costs include:

  • inventory service costs - insurance and taxes;
  • storage space costs - leasing costs or land rates;
  • inventory risk costs - these are costs related to pilferage, the risk of goods being kept so long that they become obsolete, the risk of damage, etc; and
  • carrying costs - the cost of storing - labour, depreciation and other overheads.

Inventory value

The concept of value has shifted. In recent years the concept of value has become accepted as the difference between the value a customer attributes to a product or service and the cost of acquiring value. Excessive stock holding is not only a risk in emergencies in the event of an evacuation but also not cost effective when millions are tied up in dormant stocks that may not all be utilised within reasonable time, or used at all due to rapidly changing needs. Monitoring and collaborating closely with programs on distribution rates would help in balancing the benefits.

Order management costs

Order management costs include those for issuing and closing orders, the related handling costs, and the associated communications costs. It would be prudent to benchmark these and keep them under close monitoring to ensure that service delivery is cost effective.

Cost of waste

The cost of waste covers the cost of disposing of packaging and damaged or unserviceable equipment. Waste disposal costs have sharply increased due to environmental impacts.  This aspect is covered under Reverse Logistics.

Reporting performance

Customers provide feedback on the performance of procurement. This feedback should be both qualitative and quantitative. Qualitative - how they felt about the service they were given and how helpful people in logistics were. Quantitative - is objective and measurable. This can be achieved by setting and agreeing service standards with customers, for example the time it will take for a requisition to be processed. The customer can provide how well this was met.

Information and data can be recorded and kept within logistics. The analysis of the information will provide feedback on performance. It is possible to measure performance in carrying out the logistics process particularly if there are standards set. 

For example:

  • documents sent to accounts in time;
  • goods delivered on the specified date or within the specified period of time;
  • number of times a vendor has delivered the correct goods or how many times goods have been rejected; and
  • number of requests rejected by procurement due to poor specifications.

In an emergency situation performance monitoring is a very important aspect and should be instantaneous with immediate remedial measures taken. The reporting back should be more structured and targeted to get immediate attention and action taken.

Some of the key indicators would be:

  • on time delivery;
  • delivery of exact specification requested;
  • reliable transport services; and
  • delivery of exact quantities requested.

Key Performance Indicators  management tools

Key Performance Indicators (KPIs) can be defined as specific metrics used to monitor performance on an ongoing basis.
KPIs can only be useful if the metrics selected are targeted to achieving the organisations logistics objectives. Some examples of KPIs are:

  • percentage of Items returned/rejected;
  • total dollar value of damaged/lost goods;
  • percentage of on-time deliveries;
  • inventory levels vs. forecasted need; and
  • field distribution performance.

See a sample of Transporter performance table.

Cost and performance

We use the term right cost rather than right price, as the price on its own does not reflect how much particular goods or services will cost the organisation. The aim of logistics should be to look at the total cost of obtaining the goods or services and any ongoing costs of operation or maintenance. What may seem a good initial purchase price may become a poor total cost because the supplier may add extra costs for packing or transporting the goods. For other items, such as vehicles, a good initial purchase price may become a poor total cost with expensive operating or maintenance costs.

What is the right cost for particular goods or services?

This is not an easy question to answer, as there will be a lot of different factors and constraints that will affect the initial purchase price and the total cost. The aim should be to achieve the best value for money given the different factors and constraints. Some aspects to consider:

  • actual market price;
  • other delivery conditions; and
  • source of product and extended distribution costs.

The "best price" will therefore be the most cost effective price that will ensure the goods or services are delivered in time, at the right location, in the right condition, in the right quantity and meets the stakeholders (donors/internal user/beneficiaries/public) service expectations.

Conclusion

Monitoring and evaluation are key cross-cutting components of an effective logistics process. Together they provide the management tools necessary to measure the ability of an organisation to provide the right products or service, at the right place at the right time for the right price against consistent targets set by the organisation. They provide the means to identify problems so that they can be corrected.