How many times have you heard this?.... Citrix is slow.

It has to be one of the biggest complaints I hear about Citrix, slow logons, sluggish sessions with typing delays, poor video and sound. This session is all about performance considerations when using Citrix along with some best practices to prevent steady state related performance bottlenecks.

There was a technician at the site who said it crashed immediately and a window stating 'citrix has crashed' (or simila) appeared. It didn´t help to reinstall Citrix online plugin 11.2 as we use, even after clearing out the users profile files related to citrix. Others tried to use citrix on his computer but it. Send it to desktop to check their connectivity, WiFi extenders and such as you mentioned. Then send a chart from control up with their 800ms latency over the past week and close the ticket. Train your helpdesk and desktop support teams to recognize the issue and “resolve” before it gets to you.

1. August 17, 2013 August 19, 2013 Dale Scriven 10 Comments on Slow logon’s to Citrix Web Interface or Storefront If you’ve ever been frustrated by the length of time it takes to get past the initial pre-logon screen on Citrix Web Interface or Storefront then a simple change can rectify this issue.
2. Internet Explorer 11: The VDA-side IE11 browser viewport is redirected and rendered on the client-side using the client-side installed IE11 and the Citrix Workspace app for Windows process HdxBrowser.exe. A BHO (Browser Helper Object) called CitrixHDXJsInjector is added by the VDA installer to IE11 on the VDA.
3. As Citrix continues to make further innovations in this area, there are some features added within the existing product versions that are frequently overlooked or misconfigured that can greatly improve current graphics related performance in certain environments.

I have a design methodology that simply states that performance can be guaranteed with good design and to me this is simple, never share and never over allocate backend hardware resources and providing you do your math right in your capacity planning and you stick to the best practices covered in this paper you can almost guarantee a fast desktop that in many cases one that will outperform a local PC. Try it…

What are the cause’s slowness?

There are so many things that can cause a Citrix session to slow down but the most obvious are covered here:

• Lack of capacity planning
• Using shared resources
• Session Sharing
• Hypervisor Choices
• Not having any spare capacity
• Active Directory Domain considerations
• Network Bottlenecks
• File Server locations
• Profile Management
• Too many startup processes
• Poor Anti-Virus settings
• Badly set HDX policies
• TCP Offload (under certain circumstances)

Let’s look at each item in isolation

Capacity Planning

There is no black art when it comes to Citrix Capacity Planning, it is in fact very simple math. Of course it does depend on what you are doing with Citrix as the numbers will vary greatly but I am talking about the masses here. I won’t be covering GPU or High end power users, those I will cover in a later paper.

First a few golden rules:

1. Do not place end user sessions or desktops on equipment that is shared with other systems where you cannot ring-fence the required resources, examples being a shared SAN on VMWare*.
2. Never use memory or CPU ballooning
3. Don’t bother splitting up the hard drives unless they can be spread across different dedicated LUNS, ideally you won’t be using a SAN for the Citrix Desktop (Covered later in this paper)
4. Make sure the File Servers, Print Servers and user profiles are in the same subnet as the Citrix servers, especially when using profile redirection settings
5. Reduce the logon time to be as fast as can be because the servers is under its greatest load during logonlogoff

* Exceptions considered here are if you using products like Atlantis ILIO or Nimble where disk IOPS are measured in the hundreds of thousands.

Here are the magic numbers that you need to remember:

If you can allocate theses resources to your Citrix sessions I will guarantee it will pretty fast.

SBCXenApp Sessions (Desktop and App)

• 2GB RAM per session server and 200MB for each session
• No more than 14 users per CPU Core with HT considered
• 5 Disk IOPS per session
• 100 IOPS during logon
• 100 IOPS during Logoff

VDIXenDesktop Desktop (Win 7 or above)

• 2GB RAM per
• No more than 4 users per CPU Core with HT considered
• 25 Disk IOPS per session
• 200 IOPS during logon
• 200 IOPS during Logoff

What does this all mean?

Citrix Workspace Va

In order to make sense of this we need to translate this down to physical hardware and for the sake of simplicity here I am going to take an average run of the mill mid-range HP Server, say a DL380 Gen 9 with 2 x ten core 2Ghz processors, 128GB RAM with an additional 4 x 1GB NIC along with 8 x 15K 300Gb SAS drives and 2GB Cache RAID Controllers at the cost of around £6,500 What does this give me?

• 128GB RAM (Obvious)
• 40 CPU (with Hyper Threading)
• 1440 Disk IOPS in a RAID 1+0 setup
• 1TB of usable local storage space
• 16GB of network traffic at Full Duplex

What this all comes down to is how many desktopssessions can I run per physical servers?

Let’s work it out:

As you can see from the chart above Disk IOPS are our biggest bottleneck, not CPU and Memory.

So If you’re running a XenAppSBCRDSH solution we have found that the optimum hardware configuration for each VM is 4 vCPU’s, 12GB RAM, 100GB HDD with a single NIC that is part of multi-NIC bonded network of four NICs. On this VM you you will comfortably get 35 users sessions running concurrently and on each physical server of the specification above will run up of eight of these VM’s on the same server giving you a total of 280 sessions per HP Servers but for peace of mind size it up with 200 user session in mind, knowing there is spare capacity with will be important later. User logons are critical metric also to consider and these servers are capable of handling 14 simultaneous logons at 100 IOPS with a 1440 IOPS capacity

Making the Hardware Cost Per XenApp Desktop = £32.50 per user

Let compare this with VDIXenDesktopVMWare View the numbers will be different because we are publishing many more operating System but this time we will be running Window 78 each with 2GB RAM, 1 vCPU and 25 IOPS and as we can see from the chart above you theoretically get 58 desktops running off one server but then apply our 1 third contingency we are looking realistically to achieve around 40 Desktops per physical server

Making the Hardware Cost Per VDI Desktop = £90 per user (quite a difference)

The choice to select what desktop to run VDI vs SBC will be covered in a separate paper.

So there you have it, for SBCXenApp Desktops you can each server will cope with 200 desktops and for VDI that number comes down to 40.

Using shared resources

In order to guarantee a fast desktop you need to be able to be in control of your back end resources. So many times I have seen customers who have invested heavily on great equipment like NetApp FAS units or Dell EqualLogic believing all of their storage related issues will be resolved. The problem with these types of shared resources are what they are, that is they are shared. If you are running the rest of your systems off this shared storage like SQL, Exchange or file sharing then you cannot guarantee the performance.

Look instead to utilize local storage for the desktop provision but keep the critical core services like the controllers on the shared SAN as these generally do not affect the performance.

Hypervisor Choices

Choosing the right Hypervisor to run Citrix on is very important. There are two levels of virtualisations that we need to think about here. One being the server operating system itself and the second being the virtual desktop, so we are running a virtual desktop on a virtual server and some things can be lost in translation, but about that later. The choice to virtualise Citrix is an old debate today as hardware performance has massively increased over the last few years it makes so much sense to virtualise Citrix today, though there was a time in my life I was dead against it.

There are only really three options when it comes to choosing the virtualization platform to run Citrix on, VMWare, Hyper-V and XenServer. Each of them have their merits, but as a rule I always recommend that the platform of choice for virtual desktops should be Citrix XenServer over VMWare and Hyper-V and the reason being is simple. VMWare and Hyper-V lend themselves very well for over allocation and both of them do that very well achieving excellent consolidation ratios oh and they cost money… Citrix XenServer utilises the Hyper Visor built in to modern CPU’s using a technology called Para-Virtualisation instead of the billions of line of code used with VMWare and Hyper-V which almost bare metal performance and it is free with XenDesktop. There is more to be said about this subject but this will be covered in more details in a separate paper at a later stage. Just leave comments if you have any questions.

Session Sharing

When publishing applications with XenApp enable Session Sharing this means that every user will only use one session which will reduce the load on the back end and improve the performance for the users as they don’t need to open up multiple sessions.

Make sure all applications are available across all servers using App-V where necessary and publish every application with the same sessions settings, i.e. colour depth and sound settings that way the user will always launch applications on their current session and not be logged on to multiple servers and they won’t need to go through the whole logon process every time they launch an application.

Spare Capacity

Where possible, don’t run your equipment to it maximum capacity, try to give yourself a third capacity as contingency on standby to cope with system failures and planned maintenance work. An example being if you are servicing a 900 end user virtual desktop estate build the backend solution to cope with up to 1,200 users (i.e. six session hypervisor servers instead five)

This will make the ongoing maintenance of the estate much easier and provide swing equipment when doing upgrades and maintenance tasks.

Active Directory Domain considerations

I do hate to sometimes mention the obvious but I have seen this issue too many times not mention it here. Make sure Active Directory Sites and Services is set up right with the right subnets defined and that the subnet that your Citrix servers are in are assigned to the right subnet to ensure authentication is local and not left to a random choice or worse still over a WAN connection as this will slow down logons.

Network Bottlenecks

The amount of bandwidth required for end user Citrix users is very low but on the backend the servers will require ample bandwidth and here a few guidelines I would advise you to stick to:

• Use bonded networks for general network traffic, ideally four NICS per server setup with ActiveActive LACP
• Separate the Provisioning Services on a separate VLAN again in bonded LACP Channels
• Ensure the Switch backplane has a minimum of 32GBSec Throughput
• Use Layer 3 switches and use these switches as the Gateway
• Use Citrix Netscalers for internal load balancing

File Server Considerations

There a few things to consider with your file servers with Citrix that could affect performance:

• Make sure that the file servers that are being used for Citrix Profiles and redirection are in the same subnet as the Citrix Session servers to ensure they don’t need to traverse a gateway.
• Consider the amount of storage you need and the IOPS per session especially when re-directing
• Use monitoring to keep an eye on disk queue lengths and set up alerting if the queue lengths go above 1.5 as that will significantly affect performance
• When using DFS make sure the DFS servers subnets are defined in Active Directory Sites and Services

Profile Management

Profile management is a really big subject which will be covered in a later paper. The speed of logon will dramatically effect Citrix performance due to the excessive load that logging on places on system. Here are some best practices:

• Use a Mandatory profile for all users stripped off all unnecessary junk which can be reduced in size to around 256K. This makes for an almost instant logon
• Hardcode in redirection directly in the Mandatory profile
• Use drive letters for redirection do not use UNC paths to eliminate SMB limitations
• Make sure profiles are cached locally on the Citrix servers and deleted at log off.

Too many startup processes

This is just a basic rule that gets overlooked too often. Eliminate all unnecessary startup processes run when a user logs on:

• Delete everything in the All UsersStartup Folder
• Delete all entries in the registry key HKLMSoftwareMicrosoftWindowsCurrentVersionRun

Poor Anti-Virus settings

Getting the Anti Virus settings wrong can severely affect the speed of the Citrix servers. Follow these rules to give a AV a fighting chance without killing performance:

• On-Access Scan set Only on writes
• Sensitivity set Heuristic
• Scheduled Full Scans on Session Hosts
• I highly Recommended these exclusions are set:
  • Windowssystem32spoolsv.exe
  • Windowssystem32csrss.exe
  • Windowssystem32winlogon.exe
  • Windowssystem32userinit.exe
  • Windowssystem32smss.exe
  • Program FilesCitrixGroup PolicyClient-Side ExtensionCitrixCseEngine.exe
  • Program Files (x86)CitrixSystem32wfshell.exe
  • Program Files (x86)Citrixsystem32CpSvc.exe
  • Program Files (x86)CitrixSystem32CtxSvcHost.exe
  • Program Files (x86)Citrixsystem32mfcom.exe
  • Program Files (x86)CitrixSystem32CitrixImaImaSrv.exe
  • Program Files (x86)CitrixSystem32CitrixImaIMAAdvanceSrv.exe
  • Program Files (x86)CitrixHealthMonHCAService.exe
  • Program Files (x86)CitrixStreaming ClientRadeSvc.exe
  • Program Files (x86)CitrixStreaming ClientRadeHlprSvc.exe
  • Program Files (x86)CitrixXTEbinXTE.exe
  • Program FilesCitrixIndependent Management ArchitectureRadeOffline.mdb
  • %AppData%ICAClientCache (if using pass-through authentication)

HDX policies

Citrix High Definition Experience policies can make a large number Improvements that can really improve the virtual desktop experience from redirecting video and Flash to be rendered on the local device for desktop like performance to USB and Microphone optimizations as well as providing additional tools for Microsoft Lync. There is no better virtual desktop platform than Citrix if you are looking to use Lync. Here are my recommended general guidelines:

• Configure HDX MediaStream Flash Redirection - HDX MediaStream Flash Redirection allows you to move the processing of most Adobe Flash content from Internet Explorer on the server to LAN- and WAN-connected users' Windows and Linux devices.
• Configure Audio - You configure audio through the Policies node of Citrix Studio and You control the followings settings for the audio features through the Citrix User Policy settings:
  • Audio Plug-n-Play (XenApp only)
  • Audio quality
  • Client audio redirection
  • Client microphone redirection
  • Audio redirection bandwidth limit
  • Audio redirection bandwidth limit percent
  • Audio over UDP Real-timeTransport (XenDesktop only)
  • Audio UDP Port Range (XenDesktop only)
• Configure Video Conferencing with HDX RealTime Webcam Video Compression
• Configure HDX RealTime to provide your users with a complete desktop multimedia conferencing feature.
• Configure HDX 3D - HDX 3D allows graphics-heavy applications running on XenApp to render on the server's graphics processing unit (GPU). By moving DirectX, Direct3D and Windows Presentation Foundation (WPF) rendering to the server's GPU
• Enable XenApp 6.5 OpenGL GPU Sharing Feature Add-on - This feature add-on to XenApp 6.5 enables graphics processing unit (GPU) hardware rendering of OpenGL applications in Remote Desktop sessions
• Assigning Priorities to Network Traffic - With XenApp and XenDesktop, priorities are assigned to network traffic across multiple connections for a session with quality of service (QoS)-supported routers.
• Add Dynamic Windows Preview Support - With the Dynamic Windows Preview feature enabled, the following Windows Aero preview options are available to XenApp users with published applications:
  • Taskbar Preview - In a single-monitor configuration, when the cursor hovers over a window's taskbar icon, an image of that window appears above the taskbar
  • Windows Peek - When the cursor hovers over a taskbar preview image, a full-sized image of the window appears on the screen
  • Flip - When the user presses ALT+TAB, small preview icons are shown for each open window.
  • Flip 3D - When the user presses TAB+Windows logo key, large images of the open windows cascade across the screen.
• Configuring Read-Only Access to Mapped Client Drives - With the Citrix User Policy setting Read-only client drive access, you can control whether users can copy files from their virtual environments to their user devices.

TCP Offload

You may not have heard of this one, but it has caused so many problems I am going to mention it here and that is problems with TOES cards.

What is a TOE Card?

A TOE card is a network adapter that has a built in TCP Offload Engine (hence the name TOE) and pretty much every server today will have TOE cards. Now these are great in principle as they can really improve network performance by taking some of the load from the operating system but they can sometimes cause issues on virtual machine, let me give you an example.

I had a problem at a major global bank I was working for rolling out XenApp, all went fine until we got to Tokyo. I could not get the systems in Tokyo to ping the Citrix servers hosted in London but they could ping any other servers in London, just the Citrix servers. My immediate reaction was it had to be a firewall right? After a bit of too and froing I had to acknowledge it was a Citrix issue so my trusty tool called Wireshark was brought in to action and what it showed me was that packets were being re-transmitted thousands of time and eventually dropped. What was going on? After a bit of digging I discovered that this bank had reduced their WAN MTU for the GRE tunnel they were running to Tokyo which is a perfectly normal and recommended thing to do for GRE however because the TCP Offload was trying to negotiate the MTU size which cause thousands of re-transmits the packets never made it to the Citrix session Servers. This is an example how things can get lost in translation from physical to virtual. I was running these Citrix Servers on a VMWare cluster and once the VMWare tools were installed they will detect the TOE card and attempt to use it as it did on this occasion so we had a virtual NIC and a Physical NIC both attempting to use the TCP offload engine which failed. Disabling the TCP offload on the VM instantly fixed the issue. I have also seen slow network issues and again disabling the offload engine has rectified the problem.

Summary

I hope you find this guide useful, it is not exhaustive but it will cover most of your Citrix related performance problems and you will find that they will come down to one two things, [1 stupidity] that being where the servers are not sized up properly or the file server that everyone uses is on a 10MB WAN link 100 miles away or user profiles have bloated to over 100MB etc.. [2] Stuff gets lost in translation from Physical to virtual. The conversion from traditional PC’s to virtual desktops introduces new IT challenges that unless you are experienced in you not have considered like profiles and printing.

Final thought, remember there are the P’s that you need to get right in Citrix, Performance, Printing and Profiles. Get these right and you will have a happy user base.

Citrix Workspace Slow

Brett Loveday

The COVID-19 pandemic has disrupted how every business on the planet works. Remote working has become the new normal, and IT needs to adapt quickly to support this somewhat new way of working.

I say new way of working, but it isn’t actually new at all in the grand scheme of things.

People have been working outside of the office, such as from home or other public places, for many years now. Organizations such as Citrix have provided technologies that have helped with achieving this, and many times in the IT industry we hear how the office is now just another place people can stop by to work from.

However, working remotely may now be for many of us the new normal. While there are organizations that previously had staff working from home one or two days a week, these organizations may now have staff working from home full time or the majority of their working week.

On the other hand, some organizations did not have staff working remotely at all. These organizations have had to adapt quickly because of the pandemic, and that may have been a rushed, painful process especially if the organization did not initially have the correct infrastructure in place to allow effective remote working.

Organizations now have staff working from home all over the country, and staff depending on remote technology could be using either their personal devices, or corporate laptops. To support this shift, Citrix Gateway could be configured within the organization’s private or public cloud to allow for secure remote access to virtual applications and desktops. With Citrix Gateway acting as an ICA proxy, staff may be allowed by the organization to use their personal computers, laptops, or tablets such as an iPad.

Citrix Workspace Slowest

Other organizations may use the Citrix Gateway for VPN access, and have staff again use their corporate devices, or personal devices subject to posture checking.

IT loses visibility and control

These technologies such as VPN and ICA proxy are great at allowing efficient remote working, however there are more external factors introduced with remote working that can impact the end-user experience. We now have staff connecting into our datacenters from a range of different internet connections, from 3G, 4G, Wi-Fi hotspots and broadband. This is something corporate IT can’t control.

Further to this, if staff are likely using their personal devices to connect in, those devices in some instances could be the latest and greatest Windows 10 or Apple devices that are fully patched running the latest versions of web browsers and Citrix Workspace app. This is often only a very small percentage of the userbase though, as most devices will run a mixture of operating systems from Windows 10 to Windows 7, and older macOS versions that have not been updated in 3-4 years. I often find these devices also running unsupported versions of Citrix Receiver that simply haven’t been updated since they were initially installed.

Staff still expect that if they can connect to their virtual apps and desktops, that the performance they receive is similar to what they were used to when working from the office. That is, minimal lag, a responsive application that also launches quickly, and is available whenever they need it.

The rise of multimedia

Audio and video technologies such as Zoom, and Teams have become much more popular due to the pandemic and staff working remotely. People need to stay in touch daily, from having meetings regarding ongoing projects or simply keeping in touch to prevent feeling isolated.

Usage of Teams has more than doubled in recent months, with now over 70 million active users using it around the world. Bandwidth consumption can range from 30kbps to over 1Mbps depending on if you are audio calling or in a group video call.

It is important to keep in mind that these multimedia platforms do have the ability to consume more of your available bandwidth, and for those staff members who have lower quality broadband in the home, there could be bandwidth contention with your virtual apps and desktop sessions. I should also mention the fact that if you have other members in your household streaming or playing online games from a range of devices, those activities will further add to the bandwidth contention of your home broadband.

The remote worker, returning to the office

Since the initial months of the pandemic, some restrictions are being lifted across different countries and people are allowed to return back to the office, though often in limited numbers.

Some staff are also returning to the office on a part-time basis, but the change of toolset has created a potential new problem. All these multimedia applications such as Zoom and Microsoft Teams that were adopted for keeping connected, will continue to be used even as staff return to the office.

Because of this, what starts to happen is that gradually a much higher demand is placed on the office network. The more people that return, the higher the bandwidth utilization is likely to be, higher than ever before.

Users will still also obviously continue to launch their virtual apps and desktops whilst in the office, but with the much higher bandwidth contention with peers, the ICA channels could be impacted, causing slow application launches, slow in-session performance and so on.

Ultimately, the risk of tickets reaching the desk for “Citrix is slow” problems may be more common.

Pinpointing the slowness

From an IT perspective, Citrix admins need to be able to pinpoint where the slowness is coming from when an end-user reports that “Citrix is slow” from the office or remote. Is it the home network the user is connected from, the user’s personal device, the entire corporate office, or something in the datacenter with the Citrix VDAs?

For example, it could be the end-users home network due to household members using Wi-Fi for streaming and playing video games, leading to bandwidth contention which is impacting the end-users ICA session to their virtual apps and desktops.

On the other hand, it could be that the end-user’s personal device is running poorly. Maybe it is running on quite old hardware, or there has been a lot of third-party software installed on the device which is causing high RAM or CPU utilisation. Such problems can impact the launching and in-session ICA performance, however the end-user may report to the helpdesk that Citrix is slow, not knowing that it is caused by their own device. This leads IT down the wrong path from the beginning.

Also, staff may now be reporting slowness after a return to the office, with the usage of Zoom and Teams still high, because the network infrastructure was not designed for the larger amount of traffic end-users are now pushing through it.

IT need purpose built monitoring tools to help pinpoint where the issue lies, and to quickly rule out if the issue is not with Citrix at all.

How ICA channel, network and latency monitoring can help

A purpose-built monitoring tool that can monitor each of the individual ICA channels, the end-user’s network and latency is a powerful solution to have at your disposal. Goliath Performance Monitor for example has such ability to monitor each channel and you can be selective about what channel to investigate when troubleshooting:

In the below example, an end-user reported that their session became slow suddenly and wanted to know what happened. Upon review, we can see that the end-user experienced a sudden spike in network latency, which caused the ICA latency to jump up.

At the same time, we were able to see that the user’s connection speed dropped simultaneously, which we can correlate to the sudden spike in network latency. Obviously, something happened on the end-users network at that time to cause a drop, which could have been a burst in demand for available bandwidth or just temporary slowness from any of the hops coming into the organization’s datacenter.

Finally, in this example, an end-user similar to the scenario before encountered a sudden spike in ICA RTT latency, but this time it was not actually anything user-side which caused the problem.

Upon comparing the VDA CPU utilization the end-user was connected to at the time, we can see the CPU usage increase for a period of time that matches when the ICA RTT latency jumped. This eliminates any doubt over the end-user connection, which can also be tracked under Network Latency and appears to be very stable throughout.

Summary

Remote working, and higher usage of multimedia solutions are here to stay. What that also means is troubleshooting slow session and end-user experience issues is not going away any time soon.

Luckily, there are solutions out there that can help assist and guide Citrix admins to typical “Citrix is slow” resolutions so that you don’t spend any extra time troubleshooting, especially when you cannot see the full picture for those users remotely connecting from various internet links and from personal devices.

As you can see from these examples, with actual data driven proof you can quickly isolate root cause, have your users back up and running quicker and spend less time troubleshooting end-user problems.

To learn more about how to troubleshoot “Citrix is Slow” by understanding data from ICA/HDX protocol, check out this Technical Guide to Citrix ICA/HDX sponsored by Goliath Technologies.