Monitoring and Evaluation
Not synchronized for offline browsing This Chapter is available for offline browsing
The Logistics function in humanitarian organisations is made up of people, processes and systems working together to support efficient and effective delivery of services. Controls are normally put in place to monitor weaknesses, poor designs in projects and improper implementation of programs. Based on continuous monitoring, these weaknesses or shortfalls against targets or objectives set can be corrected or revised in order to continually improve performance, thus reducing the risk of exposure and strengthening the response to needs.
Monitoring and evaluation are integral parts of management and provide a link between planning and implementation. While monitoring focuses on the activities and outputs, evaluation focuses on the outcome and goals.
Monitoring is initiated at the beginning of a programme, project or emergency response and built into the design, assessment and planning phases of the logistics aspect. It focuses on inputs and outputs and basically tracks and assesses implementation of the logistics aspect of the programme, project or emergency response. It is the continuous process of gathering logistics and programme information to measure against pre-set key performance indicators (KPI's), benchmarks or previously base-lined indicators that are aligned to the goals and objectives of the program.
Evaluation, like monitoring, is a continuous process. The evaluation of the quality of the output should be undertaken in such a way that shortcomings can be identified and corrected. Evaluation should also feed into the planning process continuously so that the planned method of the intervention can be modified to take into account the realities and conditions on the ground. Evaluation provides a tool for management to ensure that focus is maintained.
For logisticians, monitoring and evaluation may be defined as follows:
Monitoring: to review on a continuous basis the degree to which the logistics activity is completed and if targets are being met. This allows corrective actions to be taken.
Evaluation: to analyse progress towards meeting established objectives and goals. It is done on an ad hoc, monthly, quarterly or yearly basis. Evaluation provides feedback on whether plans have been met and the reasons for success or failure. It should also provide direction for future plans.
Together monitoring and evaluation provide the basis for performance management. Also key to performance management is aligning performance metrics to the goals and objectives of the program.
Without alignment, monitoring and evaluation is ineffective as the direction in which the program is heading is unknown and therefore M&E is unfocused and potentially disruptive to the program.
Without alignment you do not know what to monitor or how to evaluate it.
Monitoring and evaluation has several purposes:
Quantitative and qualitative measures can be monitored for the entire supply chain, from the manufacturer to the beneficiary, as well as for individual parts of the supply chain. Every link in the supply chain should be seen as a service receiver (from suppliers) as well as a service provider (to users). The management of stocks and the quality of services received will affect the quality of services provided. It is important to make this distinction when there is a requirement to improve service provided to the final user or consumer.
For the purpose of this document, a customer is the final user or the identified final beneficiary of the logistics service. There are two sets of customers: internal and external customers. The internal customer is the user or beneficiary department within the same organisation and the external customer are the beneficiaries in the communities that we serve.
The Monitoring and Evaluation of Services
There are two main points to consider when monitoring and evaluating order management:
Services provided to customers
This point can be looked at from the perspective of the order management team providing services to customers/users/beneficiaries, as well as the customer's perspective of the effectiveness of services received from the Order Management team. The quality of service provided to customers needs to be measured and monitored as a basis for management and planning. The most used indicator in this regard is the percentage of orders delivered in full, on time and error free (DIFOT).
The imperative for getting it right
While suitable performance measures provide the foundation for informed decision-making, improper performance measures can distort the conclusions drawn and badly impact efficiency by effectively disguising critical issues and warning signs. 'Good' performance measures have several distinguishing characteristics:
Additional key indicators:
Customer perception of service
Logisticians have priorities in their day-to-day work, however it is important to keep in mind the objective: to provide excellent service to the customer, with consideration to their objectives.
An important aspect of measuring the customer's view of the services provided is through a customer satisfaction survey. This is done in the form of a questionnaire. Indicators that are quantitative and objective monitor "hard facts" such as inventory availability and lead times. The purpose of requesting feedback from customers is to gather subjective information that cannot be measured. Customers are asked to rate the importance and quality of services received to determine their perception of logistics services (qualitative analysis).
The Customer Satisfaction Survey Questionnaire
A useful tool that can be used by the logistician is the Customer Satisfaction Survey Questionnaire. It can be used to:
Services received from suppliers (including upstream logistics centres)
The same measures can be used as for the "service provided". The main difference being that the Logistician becomes a customer of logistics services.
Performance measures provide a management framework, facilitate communication, direct behaviours within the organisation, foster improvement and assess positioning and operational capacity.
There are a great number of indicators for measuring the quality of services along the supply chain. These include time, scope, service and cost. No single indicator allows measuring all aspects of supply chain management. They should be interpreted and used in combination.
Performance of the supply chain should be measured regularly, consistently and systematically rather than waiting until services have deteriorated below acceptable levels. Monitoring should be a key element of running a logistics service.
Actual performance should be compared with the goals and objectives established for the program, which defines the level of service logistics tries to achieve.
Some aspects to monitor in logistics
Supply chain response/lead time
Lead time is the time between placing an order and receiving the goods or service. Delivery too early or too late may also incur unnecessary costs. Delivery too early can mean goods have to be stored until needed and will incur costs whilst being stored in warehouses. Delivery too late can mean the costs of setting up facilities, for example feeding stations and having people ready to distribute goods, is wasted because the goods have not been delivered. It can cause the organisation to incur additional transport costs, for example, aircraft/helicopters have to be used to move the goods more quickly along the next part of the supply chain. In disaster/emergency relief situations, timing of delivery can have a serious impact on the relief operation and on the beneficiaries.
Information on status of orders
The internal performance of the logistics function is dependent on the efficiency and effectiveness of each of the logistics components. For example, one performance indicator for procurement, would be its ability to disseminate information on the number of orders issued. This will enable the warehouse to provision for storage space. Unexpected deliveries can disrupt operations and put the stock at risk of being stolen when left in the open.
The measurement of efficiency is sometimes relative and dependant on what an entity defines as efficiency. In this context, efficiency is the satisfactory delivery of a logistics service that enables the end user to fulfil the intended purpose of the request. A good example is the request for malaria prevention medication ordered to be pre-positioned before a malaria season. A late delivery would mean higher incidents of malaria and an increase in the request for malaria treatment rather than malaria prevention drug.
Total supply chain costs
The total cost concept focuses on reducing the total cost of logistics rather than the cost of each activity. An organisation should monitor the cost reduction across the board and evaluate the impact on each of the logistics components. For example, purchasing in bulk may reduce the cost of the product but at the same time increase the stock holding costs.
Inventory carrying costs include:
The concept of value has shifted. In recent years the concept of value has become accepted as the difference between the value a customer attributes to a product or service and the cost of acquiring value. Excessive stock holding is not only a risk in emergencies in the event of an evacuation but also not cost effective when millions are tied up in dormant stocks that may not all be utilised within reasonable time, or used at all due to rapidly changing needs. Monitoring and collaborating closely with programs on distribution rates would help in balancing the benefits.
Order management costs
Order management costs include those for issuing and closing orders, the related handling costs, and the associated communications costs. It would be prudent to benchmark these and keep them under close monitoring to ensure that service delivery is cost effective.
Cost of waste
The cost of waste covers the cost of disposing of packaging and damaged or unserviceable equipment. Waste disposal costs have sharply increased due to environmental impacts. This aspect is covered under Reverse Logistics.
Customers provide feedback on the performance of procurement. This feedback should be both qualitative and quantitative. Qualitative - how they felt about the service they were given and how helpful people in logistics were. Quantitative - is objective and measurable. This can be achieved by setting and agreeing service standards with customers, for example the time it will take for a requisition to be processed. The customer can provide how well this was met.
Information and data can be recorded and kept within logistics. The analysis of the information will provide feedback on performance. It is possible to measure performance in carrying out the logistics process particularly if there are standards set.
In an emergency situation performance monitoring is a very important aspect and should be instantaneous with immediate remedial measures taken. The reporting back should be more structured and targeted to get immediate attention and action taken.
Some of the key indicators would be:
Key Performance Indicators management tools
Key Performance Indicators (KPIs) can be defined as specific metrics used to monitor performance on an ongoing basis.
See a sample of Transporter performance table.
Cost and performance
We use the term right cost rather than right price, as the price on its own does not reflect how much particular goods or services will cost the organisation. The aim of logistics should be to look at the total cost of obtaining the goods or services and any ongoing costs of operation or maintenance. What may seem a good initial purchase price may become a poor total cost because the supplier may add extra costs for packing or transporting the goods. For other items, such as vehicles, a good initial purchase price may become a poor total cost with expensive operating or maintenance costs.
What is the right cost for particular goods or services?
This is not an easy question to answer, as there will be a lot of different factors and constraints that will affect the initial purchase price and the total cost. The aim should be to achieve the best value for money given the different factors and constraints. Some aspects to consider:
The "best price" will therefore be the most cost effective price that will ensure the goods or services are delivered in time, at the right location, in the right condition, in the right quantity and meets the stakeholders (donors/internal user/beneficiaries/public) service expectations.
Monitoring and evaluation are key cross-cutting components of an effective logistics process. Together they provide the management tools necessary to measure the ability of an organisation to provide the right products or service, at the right place at the right time for the right price against consistent targets set by the organisation. They provide the means to identify problems so that they can be corrected.