Logging Format

Starting with features introduced to General Availability in Version 22 Continuous Deployment version, some application and service logs are stored in a new location and format. Where logs traditionally were stored in SQL, they are now logged directly to disk using a built-in logging framework. The log directory for these logs is determined by the Confirmit.Site.LogPath variable in Octopus. As is the case with IISlogs, there may be reasons why logs need to be kept in an unchanged format for an extended time, so these logs are not cleaned up by the Horizons application. We strongly recommend that you determine your requirements for keeping logs for audit purposes, and clean them as necessary.

Some background for the change: The advantages of changing to this format are that it significantly reduces the cost of introducing rich logging to new features (thereby reducing the time for the feature to become available), it reduces strain on the back-end SQL clusters, and it reduces strain on SQL when viewing logfiles.

The structure of the logging directory will be <Confirmit.Site.LogPath>\services\<service name>\<log>.json. The <Confirmit.Site.LogPath> path is a site variable defined in Octopus.

Note: For troubleshooting purposes, Forsta Support are able to parse logs created by your installation. This will require you making them available to ForstaSupport. Compressing and encrypting logs first is recommended.

TheForsta Plus logging syntax uses standardized JSON format that can be read by various log processing products such as Splunk, Graylog, Sumo Logic or ELK. Should you have a logging system already available for other monitoring purposes, this can be used for Forsta Plus logs.

Log Processing

For a complete logging solution, a system should be used for picking up, processing and centralizing the produced logs automatically. As mentioned previously, the generated log files can be processed using various products, however Forsta has experience with and is using the ELK stack for our own test and SaaS environments. NXLog (https://nxlog.co/, an agent running on each Horizons server) monitors the log directories and writes the log-entries to Logstash for indexing in an Elasticsearch cluster. Kibana is then used for visualization of the data. Elasticsearch, Logstash and Kibana is part of an open source software stack (ELK) which can be downloaded and installed free of charge.

Note: Forsta provides an Octopus-deployable ELK-based solution for log analysis. For information on this please contact Forsta Support.

To set up an Elasticsearch cluster manually, follow the steps here: https://www.elastic.co/guide/en/elasticsearch/reference/current/windows.html. Please refer to documentation by Elastic for assistance with the ELK components. Forsta Support will be able to assist you with providing appropriate configuration to digest the logs created by Forsta Plus.

Note: For larger installations we recommend getting a support agreement in place with Elastic, or a company providing such service.

Log Types and Formats

Service log file. Expected file name: service-<date>.json

This log type is used for service-related information, such as providing metrics for when a task has been processed, containing the task type, task duration, task delay before being picked up etc.

Json FieldName (* = mandatory)	Format	Description
CorrelationId	string	Unique request identifier. Identifier of the client request used to relate log records from different sources and infer causality. Passes from the client in X-Confirmit-Correlation-Id header. If header is empty service should generate an globally unique id.
Service	string	Name of the application / service that is doing the logging. The name format is: Confirmit.X.Y. (Example: Confirmit.Dictionary.Api, Confirmit.SurveyDesigner.Client).
ReferrerService	string	Name of the client application / service. (The service calling this service). Generated by client and passed in X-Confirmit-User-Agent.
InitiatingService	string	Name of the service that initiated the request. Typically the first service will set it's own service name to InitiatingService. Passes from the client in X-Confirmit-Initiating-Service header.
EventTime*	ISO8601 with offset (e.g. 2015-11-11T12:12:00+01)	Time of the logging event.
Severity	one of Fatal/Error/Warn/Info/Debug/Trace	Level of the logging event.
Logger	Namespace+ClassName	Full type name(namespace + class name) of the caller issuing the logging request.
Message	string	Custom message.
ExceptionType	Namespace+ClassName	Full type of the exception.
Exception	string	Should be equivalent of Exception.ToString method. Exception type, message, inner exceptions, stack trace.
Username	string	Optional: The name of the user that is executing the service.
Company	string	Optional: The name of company of the user that is executing the service.
ServiceVersion	string	Optional: Version of the application / service.
ConfirmitSite*	lowercase string	Name of the site in lowercase.
SourceModuleName*	string	Logical module or service, lowercase with “-“ (e.g. “discovery-analytics”, “confirmit-identity”).
SourceModuleType*	one of service/access/iis	Ref the log format specification.
TimeTakenMicros	long, micro seconds	Optional: The time it took to process the request/task/job.
{CustomField}	*	Optional: Services can add any number of custom fields.
Custom fields used for activity logging
Activity	string	Activity name.
ResourceCompany	string	The name of the company which owns the activity performed on a resource.

Access log file: Expected file name: access-<date>.json

This log type logs service access events. This is useful for gauging traffic patterns, amount of hits to one service from another.

Json FieldName	Format	Description
CorrelationId		See service log.
Service		See service log.
ReferrerService		See service log.
InitiatingService		See service log.
EventTime		See service log.
Severity		See service log.
Logger		See service log.
Username		See service log.
Company		See service log.
ServiceVersion		See service log.
ConfirmitSite		See service log.
SourceModuleName		See service log.
SourceModuleType		See service log.
RequestUri	string	The uri of the request.
ResponseStatusCode	int, any valid http status code	The response status code of the request.
RequestHttpVerb	uppercase string, any valid http verb	The http verb of the request.
TimeTakenMicros	long, micro seconds	The time it took to process the request.
{CustomField}		See service log.

IIS Logs

The ELK system can be used to index IIS logs to give better insight into your traffic. This is most easily achieved by updating IIS to log to the same logging directory as the Forsta .Site.LogPath Octopus variable, for example in a subfolder called 'IIS'. The log fields in IIS have to match the log fields processed by Logstash in the logstash config. Forsta SaaS sites use the log fields outlined below.

Name	Field
Date	Date
Time	time
Client IP Address	c-ip
User Name	cs-username
Server IP address	s-ip
Server Port	s-port
Method	cs-method
URI Stem	cs-uri-stem
URI Query	cs-uri-query
Protocol Status	sc-status
Protocol Substatus	sc-substatus
Win32 Status	sc-win32-status
Bytes Sent	sc-bytes
Byte Received	cs-bytes
Time Taken	time-taken
Protocol Verions	cs-version
User Agent	cs(User-Agent)
Referer	cs(Referer)