Forgot your password?
typodupeerror

Using VMWare and Citrix in Tandem? 76

Posted by Cliff
Dysfnctnl85 asks: "As a follow up to the previous discussion 'Alternatives to Citrix Remote Computing?', I've hit another brick wall in my quest to enhance the way my company does remote computing. Right now I've setup Windows Server 2003 Enterprise x64 R2 on two 64-bit machines with 16gb of RAM each. Before I can setup Presentation Server 4, I need to install the Novell client to allow access to our NetWare servers. After doing some research on Google, and hopping forums on the Novell Support boards, I've determined that Novell has no plans to release a 64-bit client for any Microsoft OS until Vista launches." Has anyone managed to get VMWare, Citrix and 64-bit Windows working together?
"Now I'm sure there are other companies out there in a similar situation (as noted on the forums and Google Groups), so I then decided to look into the virtualization market to see if I can still make my dream happen. I've been emailing my Citrix rep who in turn has been speaking with a Citrix engineer who is currently training with VMWare, coincidentally. I'm wondering if anyone has successfully ran a VMWare + Citrix solution in order to fully take advantage of dual 64bit procs, a Windows 64-bit OS, and 16GB of RAM. I was thinking of running 2 Citrix Servers within VMWare to handle maybe 8GB, effectively making 4 public Citrix servers, but I'm not sure what the best solution would be."
This discussion has been archived. No new comments can be posted.

Using VMWare and Citrix in Tandem?

Comments Filter:
  • D'oh! (Score:2, Interesting)

    by sharkey (16670)
    I feel for you, buddy. I've been at the "Oh shit!" stage of realizing during implementation that I missed something $REALLY_IMPORTANT in planning myself.

    I am also hoping for some interesting and informative answers, since I am currently investigating using Windows Enterprise x64 to do a Terminal Services environment within MS Virtual Server using the "free" licensing of the virtual OSes. (My $ORK_PLACE steps on your neck for buying a 6-pack of Coke when on the road, penny-pinching *EXPLETIVE DELETED*.)
  • Huh? (Score:4, Insightful)

    by obeythefist (719316) on Tuesday July 11, 2006 @10:13PM (#15702938) Journal
    The problem seems to be Novell drivers, not anything else. Try running it without Novell connectivity first.
    • The problem seems to be Novell drivers, not anything else. Try running it without Novell connectivity first.

      Let me start by saying that I'm Dysfnctnl85's coworker. With our company's infrastructure, we *have* to have Novell access for the clients. We are using it primarily for NDPS and our NFS. All of the users' files are stored on Novell shares with a great deal of permissions settings, so there's no way of getting around needing Novell, short of migrating to another network architecture.
      • Have you looked at configuring CIFS support on the Netware servers and using the iPrint client (which replaces the need for the novell client although it utilises the NDPS backend). I cannot tell you how well (if at all) the iprint client would work in a 64bit environment, but if used as passthru devices within the session it may not be a big deal...
        • In order to run CIFS, we would need to upgrade from NetWare 5.x to 6.x, and even then the *way* users would connect to our file servers would be different. Also, CIFS does not support login scripts -- something that is essential to our environment.

          I've been told by Novell support personnel that in order to make something "like" login scripts work in CIFS, we would also need to upgrade ZenWorks to the latest version. And even then it's not the same as our existing architecture.

          In other words, accordi
          • Is upgrading to Netware 6.5 totally out of the question?

            If not what about upgrading to 6.5, enabling CIFS and using windows to run a script to map drives?

            I mean you could just make a .bat file with a bunch of net use statements to map the drives once CIFS is going.

            • Yes it is. Not only do we not want to add another six-digit figure to this year's budget, but with how things are going with Novell--especially with this little issue--they're going to have a harder time selling us the next rev of their product. If they had just a little more business sense, we might actually stay with them. It just takes the little action of keeping the customers happy. How big of a deal is it to push out a 64-bit client?
          • Sorry - I made the obviously false assumption that you were running 6.5 or greater seeing as 6.0 and previous is EOL'd... Methinks you've got interesting times ahead..
      • I see the importance of compatibility with the environment, but the "Ask slashdot" question was more along the lines of "Citrix and VMWare don't work together", which is a little misleading. I work for a sizeable, you could say world leading resource company, and we use Citrix and VMWare in our development environments on a daily basis. No Novell stuff. Maybe that was just the editor who posted the story.

        Now I know the integrated MS client for Novell is awful, but does that work at all? Or would it be p
  • by SirTalon42 (751509) on Tuesday July 11, 2006 @10:17PM (#15702955)
    You could always run a 32 bit OS on the servers (at least till novell releases the 64bit client). If theres nothing you need specifically from the 64bit-ness you won't really be losing anything really (though I don't really know what hardware or software you plan on running).
    • ram? wouldn't the limit be 4 gig if not in 64bit mode?
    • He'l lose performance, running in 32bit mode you have to use nasty paging hacks to access 16GB of ram, and i believe theres a 2GB or 4GB limit per process (which might not matter).
      Also you lose access to the extra registers available in 64bit mode...
      Finally if he's using AMD cpus, then no 32bit version of windows supports NUMA properly (not sure if the 64bit versions do)
  • We're doing it! (Score:4, Informative)

    by axp_bofh (930745) on Tuesday July 11, 2006 @10:28PM (#15703006)
    I'll try to get technical details tomorrow from the Citrix team (I'm on the VMS end of things), but we're a large healthcare system running a moderately large Citrix farm (~100 servers) for our clinical systems. We've got 4 DL-585's (IIRC) running 2k3 and six VMWare Citrix instances per server in production. User loading is about 20-25 users / "server". Once we got through some initial headaches, it's been quite solid. One very nice thing is that if a "server" gets bollixed up, we don't go through the usual Ghost re-imaging process to restore the server, but just copy over the VM disk image again.
    • "Copy over the disk image again"? Don't VMware's server products support snapshots and incremental disk writes which you could simply discard?
      • Yes, but that impacts performance and not recommended running all the time. Of couse you could turn the 'snapshot' on in realtime, do whatever you want (upgrade/install software), and then commmit those changes. But running in snapshot mode all the time, especially on a terminal server, is not recommended, and once that snapshot gets a certain size, it can take HOURS to commit when the time comes.
        • Also, running in redo mode, snapshots typically can eat up a LOT of space very quickly because it saves the entire delta of everything that goes on, file deletes, changes, everything.
  • At least, it did under XP Professional x64 w/ IIS installed, back when I was using that.

    It runs under WoW64, but seems to work fine. I did it for months without incident.

    VMWare supports x64.. but not by providing x64 binaries, just by insuring their code runs under Windows on Windows.

    <3 your f*cked up penny pinching configuration.. I used to work for a company like that, and it sucks.
    • VMWare supports x64.. but not by providing x64 binaries, just by insuring their code runs under Windows on Windows.

      Elaborating on the Parent further, WMWare Workstation (5.5.1 if anyones keeping track, not tried Server or had the luck to get grubby with ESX) does run as a 32-bit process on x64 Windows, BUT on x64 it allows you to also have x64 Guest OSs, whereas if you run it on a 32 bit host OS, you are (obviously) restricted to to 32-bit only Guest OSs.

      The thing I don't get is, how can a 32-bit Applicatio

      • The 32 bit version also allows you to run 64-bit OS'es, you're basically running 64 native, 32 emulated, 64 emulated when you run 64 bit OSes on a 64bit OS hosting vmware. :)

        It just emulates the x86-64 instruction set and uses longer 32bit operations to acheive the 64bit functionality.
  • Is Netware needed? (Score:4, Interesting)

    by mkiwi (585287) on Tuesday July 11, 2006 @10:37PM (#15703037)
    So you want to run Citrix as your front end. That is a pretty good choice.

    The main question is what are you running behind Citrix. Citrix itself does very little but present a screen to a virtual server. My experience has been that the thing behind Citrix- Netware in your case -is the thing that causes the most problems. Definately consider switching to an A/D setup if you have not already started migrating users from Netware to A/D. With A/D and cool projects like OpenLDAP you can do some pretty neat web based things with Java or PHP. (I have not checked in to other languages as of yet) When your company asks you to implement a fully customizable web portal using IIS (eek!) that accesses all this information, PHP is a good ally to have.

    • Nothing wrong with Novell's eDirectory/NDS. It is generally considered better than A/D and has the LDAP compatability layer.

      Anyway, sounds like they are pretty set on Netware for now. I think he made that pretty clear.

      -matthew
    • Definately consider switching to an A/D setup if you have not already started migrating users from Netware to A/D.

      Your use of the word "Netware" implies you're mixing up Novell Netware (the operating system) with Novell eDirectory (the directory service).

      With A/D and cool projects like OpenLDAP you can do some pretty neat web based things with Java or PHP.

      ...where as this is not possible with Novell eDirectory? I'm doing some pretty neat things with Bash and Novell eDirectory. Haven't tried it wi

    • by Bert64 (520050)
      In terms of security, A/D is actually a lot more of a headache than Novell's offering...
      But your right, the biggest issue with citrix is the applications behind it, many windows programs are simply not designed with security or multiuser usage in mind, there are often ways to execute arbitrary code and very little protection against excessive resource utilisation, create a corrupt word document, load it up on a shared citrix server and watch the calls to support go through the roof.
  • Citrix is a virtual computing environment. Users are given virtual workspaces on top of their own workspace. You're wanting to put two virtual workspace servers, inside of an already virtual environment. Doubling up layers of something aren't always a good thing. Think double nat'ing - yeah, you can access resources on the other side of your double nat, but it will always cause problems eventually.

    I was working with someone who wanted to do this very same thing recently and the answer from both myself, and Citrix was "no, what the hell is wrong with you." :-P I was also working with a Citrix engineer about a month ago who was testing out the same very thing you are talking about (stress tested to be a production environment not just "oh yeah, it boots, connects, NEXT") and his findings were basically "yes, it is possible, is it worth it? will it continue to work well? will performance be maintained?" The answers were all no. This was tested on both 32 and 64-bit environments all with large ammounts of RAM.
    • I can't say who I work for, aside from the fact that it's financial services, but something is wrong. With regards to "I was also working with a Citrix engineer about a month ago who was testing out the same very thing you are talking about," it seems your engineer is unaware of the solutions Citrix is supporting with some of its biggest and best clients.

      • Just because something is supported doesn't mean it's best practice, or even good practice. Citrix is really good about supporting a lot of things. Their mentality is "you paid for our product, we'll help you get it to work as best we can." But again, that doesn't always mean it's the best way to go... or even a good way...
        • I'm not a guru on this, but once tried VMWare 3.2 with Citrix on basic 32 bit x86s a few years back. Never did get it working well. Possibly would have better luck now, with all the improvements. Yeah, it wasn't the greatest idea, but that's the sort of stuff you're told to check out when the ruling paradigm is paranoia over security. If prompting for a password increases security, making users go thru 2 or more password prompts is even more secure, right?

          AMD and Intel's websites don't make it easy to

  • Horrible. (Score:3, Informative)

    by 222 (551054) * <<stormseeker> <at> <gmail.com>> on Tuesday July 11, 2006 @10:44PM (#15703061) Homepage
    I currently have a duel 3.4 64 bit Xeon box / 8GB RAM running VMware server beta and it's really not worth persuing. Any qualified Citrix consultant would tell you the same...I'm almost certain Citrix themselves don't recomment a setup like this.
    I know this because I'm an avid fan of VMware, and inquired myself... (I'm also currently upgrading to PS4).

    If you want to use VMware, get ESX...At least you can retain some performance, and VMotion offers nice flexibity. (At a price, though!)
    It's not to say that VMware can't play a role in a Citrix rollout...It's a fine testing platform, and also a solid choice for Installation Manager...but aside from that you're wasting your time.

    After speaking with a couple of Citrix consultants, I've used VMware exactly as I've described, and it's worked out fairly well. Virtualization is godsend, but not fit for *every* problem... yet ;)
    • It all depends, there are lots and lots of installations of Citrix inside VMware, it's a relatively common action. The main reason behind it is the memory address limitations of 32bit windows, you can create as many citrix instances as you want for no cost (each windows server is an additional cost, but in the grand scheme of citrix it's a minor cost). As for the 64bit windows + citrix I don't know how things play (I'm not really a windows guy but I know enough from being involved in their projects).

      And i
      • It's not a memory issue. Hell, most of my VM's even come close to capping their potential limit.

        As far as VM Citrix setups go, I'd love to see how that worked.... but as you said ESX is better. Comparing the ROI on Citrix VS ESX isn't an easy task, though... and while GSX may be better, I still fail to believe that it can provide a responsive Citrix config.

        I'm not refusing to believe, though... I'd just like to see it. I've been the primary pusher of VM's (the DR aspect alone is enough to give me a gumby
        • Actually it is a memory issue, with win32 you have to start doing tricks to address higher memory space (/3 /4 /pae each having their drawbacks and problems with citrix & terminal services). Citrix+32bit tends to top out and adding more cpu gives you hardly any benefits because you've run out of addressable kernel memory space. It's been used for this purpose for a number of years now.
          • I was more so referencing Citrix VM performance being unacceptable, and that memory availability was far less of an issue than virtualization itself; at least without using ESX.
  • You have no reason to go 64-bit in the corporate right now. Its still not there yet. You have almost nothing to gain from going 64-bit.

    So whats the problem again?

    • We have every reason to, purely from a terminal services perspective. The ability to address more than 8GB of RAM is almost a requirement for a decent terminal services environment. On top of that, Citrix supports 64-bit now and the performance advantageous are ridiculous.
    • If you are memory bound, adding more memory is the best way to increase capacity and/or performance.

      Win2K3 Std 32-bit supports 4GB RAM, which is fine unless you need more than 4GB.
      Win2K3 Enterprise 32-bit supports like 32GB, but you'd have to pony up for the new OS (cost = cost of RAM + cost of Win2K3)
      Win2K3 Std 64-bit supports 6GB RAM (cost = cost of RAM)

      Moving to 64-bit allows more memory usage without paying a larger M$ tax.

      Test it, though. Then, test some more.
  • by illumin8 (148082) on Tuesday July 11, 2006 @11:53PM (#15703281) Journal
    We have about 10 Citrix Servers running on a VMware ESX 2.5.3 system on HP BladeCenter (AMD Opterons). It works pretty well overall. We found out that you definitely need the SMP upgrade component so each Citrix instance can access two physical processors. VMware ESX 2.5.3 only supports 3600 MB max memory per guest OS and only 32-bit guest OS's, however, VMware ESX 3 just came out and it now supports 64-bit guests, with up to 16GB of RAM each, and up to 4 processors each (physical processors, not just virtual).

    The benefits of ESX server are pretty great. Secure remote console. Remote power. Ability to clone your VMs (with VirtualCenter, or a free perl script). Ability to migrate a running VM to a different server without shutting it down (google vmotion). The benefits of running virtualized are even greater than maximizing the use of hardware. Manageability is a big plus to going virtual.

    Anyway, in your situation, I would recommend installing ESX server on your two big boxes, and using many smaller 4GB 32-bit Citrix servers. Citrix will automatically load balance your apps among your server farm, and ESX will let you load about 4-5 Citrix servers on a single physical box/blade.

    Also, get some shared storage (SAN, or even SCSI disk shelf attached to both servers) so you can use Vmotion to migrate VMs around. Imagine how cool it is when you need to do a hardware upgrade or fix a bad component to just migrate the VMs off, do your maintenance, then migrate them back on, without scheduling downtime or the users even noticing. I've even run a countinuous ping to a VM, done a migration from one blade to another, and watched it never miss a single ping. It dumps the contents of memory across a gigabit ethernet connection to the other node, then somehow points it at the shared disk drive on storage, never missing a beat/ping...

    As always, YMMV.
    • Was there any additional latency on a couple of the ping responses?

      Your solution also works around a common problem - resource utilisation and general user fuckups, each user can only crash one of the smaller virtual boxes instead of the whole system.
      • Was there any additional latency on a couple of the ping responses?

        Yes, I have to admit that one or two of the ping responses is a little delayed during the final cutover. It first does the memory copy across gigabit, which takes about 30 seconds, then when the receiving server is ready, there is a final cutover where the VM is killed on one server and started up on the other. That is when I saw latency jump from 0.1 ms up to a couple hundred ms for a couple pings.
        • I just finished the ESX class. One of the labs was on VMotion, so I used that lab to try to "trick" VMotion into sucking.

          My laptop ----> MS terminal svcs to VM1
          My laptop ----> ping VM1

          VM1 ----> ping VM2
          VM1 ----> ping my laptop
          VM1 ----> terminal session to VM2

          VM2 ----> ping VM1

          When I did the Vmotion, I dropped one packet from my laptop to VM1. The other continuous pings didn't drop a single packet. The terminal session from my laptop to VM1 hitched for a half-second (I had grabbed a window
          • Wow I can't even get it to hang up for 1/2 second :) Probably whatever switches were between you and the ESX servers took longer to update their forwarding tables than mine did...
            • Well to be fair, the boxes were DL360's with only 4GB ram and another "team" was doing a Vmotion at the same time I did my testing.

              There was 1 switch connecting all the servers together - an HP Procurve (and not a high end one, from what I understand)
  • by ejoe_mac (560743) on Wednesday July 12, 2006 @01:13AM (#15703504)
    First off decide of you really need Citrix or not. There are a few things it does well, mostly on a management / printing basis. Take a look at some sort of SSL Presentation box (F5 Firepass / etc) to do your presentation. Using basic Termainal Services works fine for some situations.

    Now Microsoft is allowing 4 free instances of their OS when you're running on Windows 2003 R3 Enterprise/Advanced, and using Virtual Server 2005 R2. I know it's a MS hot dog next to VMWare's Prime Rib, but when $$ matters there is compromise to be had.

    I've used ESX for Win2003 std Terminal Server - due to the users each mapping 4 printers back each (yea Windows Server with 35 people connected, each bringing 4 printers - didn't work well). There's a check box in ESX for "Citrix Workload".

    In a perfect situation, I'd use Citrix to publish applications. I'd create 1-3 VM's on each server for each application published (5 apps = 5-15 VM's per server). Use Citrix to balance the load across those servers (or an external appliance). This would allow for a fairly consistant load across the servers without any additional features. If you're in it for the money, create 2 VM's per task and use the new Vitual Infrastructure 3 DRS feature to allow automatic VMotion if a single server gets overloaded.

    Something to think about, but remember using a Vitrual platform has so many advantages to strictly hardware I'd overlook the Citrix people saying "no". Rebuilding a server in 3 short mouse clicks is just too amazing.
  • The situation I'm in has more to do with our existing infrastructure than anything else. Terminal Services environments such as Citrix require large amounts of RAM, and moving to 64-bit allowed us to put as much RAM as we wanted in some new boxes. Now, the question is whether or not to change our existing architecture completely. Users are *accustomed* to running Citrix and published apps now, and this is the most important thing for us to consider in our new implementation. Sure, there may be overall b
    • We have a similar situation - Novell file servers + Citrix servers + Netware client = 'teh suck'. In our case, we want to provide users access to their home drives, department drives, and a large mail archive storage cluster (all on Netware). As an aside, we're ditching Novell as fast as possible. Which isn't very fast. More like glacial. Anyway...

      Dealing with the netware client was a pain and CSNW sucks for more than 1 server (users keep having to enter their -fairly long - context and tree) in an NDS tree
    • Quote:

      The "oh sh!t" moment occurred when I realized how big of a hurdle Novell would present (and for reasons that are beyond my comprehension). All of our file servers run NetWare, and to make a terminal services environment work, we need users to be able to connect to all of their files the same way they connect to them now. For this reason, it is not an option to upgrade to NetWare 6.x.

      What do you mean? "We need users to connect the same way?" I've never heard users demand anything like "I must map m

  • VMWare + Citrix presents an interesting new way of looking at providing remote access, but IMO it hooks VMWare's very strong wagon to Citrix' aging horse. IT Administrators tend to like Citrix because it gives them an easy way of centrally managing their remote users, but non-LAN-connected users hate Citrix because of the reduced graphics quality and poor performance over slow links.

    There are some other interesting solutions [roudybob.net] out there that use virtualization concepts to provide better end user experiences.
    • There's no remote access solution that beats Citrix, period. Email and files are too big to drag across dialup in most organizations today, and no remote acess protocol that I have seen is better than ICA over high latency, low bandwidth links (NX is close). Even most web based corporate apps are heavier on bandwidth requirements than a Citrix session. If you can point me to something that you think works better than Citrix I'd be interested, but skeptical.
      • There is a pretty good size market for an ICA layer for regular desktop versions of Windows. If I could install XP desktops into VM's on a central server running VMware ESX and export them to users (on terminals or repurposed pc's) via ICA you could sell it to me in less than ONE SECOND -- particularly if you want to go ahead and support multiple monitors...
        • http://www.vmware.com/solutions/desktop/vdi.html [vmware.com]

          I believe this is using some sort of citrix management, but not sure about the protocol, probably still RDP.
          • Well I guess I'll know more tomorrow when I have a conference call set up with their engineers about what we would need in order to deploy that product here. I am pretty positive it doesnt use ICA or even ship with any special management tools. From what I know so far this VDI thing is not so much of a product but more of a concept that "Hey! Don't forget that XP has an RDP server in it!" Multi-monitor support is our biggest hang up; other than that it's just a very easy way to manage a bunch of desktops wi
            • The dual monitor would be nice, I've been doing it for years with a virtual XP running on an ESX host, and running rdesktop at a really high resolution to span monitors, but it isnt perfect. Plus there isnt any dual monitor thin clients.

              I was watching a sales webinar a few weeks ago, and VDI seemed like a Citrix farm running XP machines. I dont remember if they are using the ICA protocol, if it did that would be awesome.
  • I was thinking of running 2 Citrix Servers within VMWare to handle maybe 8GB, effectively making 4 public Citrix servers, but I'm not sure what the best solution would be."

    There's a lot of overlap between VMware and Citrix to be honest. People usually have multiple citrix servers in a farm for 2 reasons. First is to handle large loads. Second is to have a testbed and make it easier for your admins. (I'm assuming you're not virtualizing to do something specialized like training classes or simulate geogr

    • Spend a little time asking yourself what those novell desktop apps do. Can they be delivered without client32? Novell is constantly pushing to pull off the desktop and into server stacks, but keeps faltering. There's enough tools to do it if you want to, but they're harder to use than the deskside tools that integrate with windows... And everyone constantly sees issues like you are where Novell lags behind Microsoft's changes to windows. (In the past I do think Microsoft did this intentionally) It sounds li

      • As an added bonus, if you have a Citrix server in a VM, disaster recovery is tons easier.

        DR w/out VMWare:

        1) Acquire new hardware
        2) Build base OS
        3) Service packs, reboot
        4) Hotfixes, reboot
        5) More hotfixes, reboot
        6) etc, reboot
        7) Use your build documentation to implement all the OS hacks needed to get that last bit of performance out of the box
        8) Install Presentation Server
        9) Citrix hotfixes, reboot
        10) More Citrix hotfixes, reboot
        11) Install your app
        12) Add any tweaks/updates to the app
        13) test and go into pr
      • That's what I thought. I wrote the first reponse this morning and I admit I didn't go read your previous AskSlashdot about VMware in general. I see what kind of environment you're running now. You're dealing with the typical "legacy can't be touched" problem. If you're still on older Netware (please tell me it's not IPX based too) you're going to keep hitting problems.

        You might attack the problem on two fronts. Virtualize your Netware servers on VMware & Virtualize the client access to the finance

  • We have a small citrix farm, with 4 real servers running the software, and one VMWare virtual server on another computer. As we only have 150 employees, load balancing isn't necessary. We've found that this will work with the version of Citrix we use (Metaframe Server XP) and the free beta VMWare. However, you will see significant performance degredation on users on the VM machine. This probably comes from VMWare running on top of the OS in the beta release. When you get the full paid for version of VM
  • At the moment, it's impossible to identify Citrix streams except in the generic sense of being Citrix. And even then, documentation is inconsistant. (TCP port 1494 is well-known, but there are also vague references to UDP port 1604, but what this does seems to be unknown.)

    Why is this important? Because Citrix is a network hog. Running multiple Citrix instances via VMWare is going to seriously kill the network card - if not the network - unless you can isolate the traffic of greatest interest. Since these ar

    • First of all, Citrix maybe uses 200k/s bursts and uses on average 18-20k/s. But, the VMWare servers will have 4 NICs anyways, so even if it WAS an issue, it's not. But it isn't. So yeah.
    • by narf (207)
      Port 1604 is unknown? Hardly. It's the older-style ICA browsing service, used by the Citrix ICA client to find server and application lists. Modern Citrix deployments should be at least using HTTP or Secure Gateway instead of using the UDP browsing (which can be very unreliable ... imagine that).

      And I can attest that at least one commercial product (PacketShaper by Packeteer) can identify individual published applications and apply shaping policies accordingly.

      And a network hog? ICA is a pretty dang light p

When someone says "I want a programming language in which I need only say what I wish done," give him a lollipop.

Working...